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Abstract 

This  research  was  conducted  to  determine  how  a  Royal 
Australian  Air  Force  technical  maintenance  publication  could 
be  transformed  into  a  task  oriented  hypermedia  application 
and  to  develop  a  model  that  could  be  used  for  future  similar 
transformations. 

The  model  was  developed  following  in-depth  analysis  of 
the  existing  USAF  F-15E  maintenance  publication  set  and 
validation  was  conducted  using  a  prototype  developed  in 
HyperWriter,  a  commercial  hypermedia  product. 

The  prototype  model  was  based  around  the  performance  of 
three  maintenance  tasks  on  an  aircraft  sub-subsystem  and 
incorporated  online  user  access  to  the  necessary  support 
data  for  task  completion.  Publication  managers,  producers, 
and  users  were  involved  in  prototype  testing  and  results 
validated  the  model  design  with  limited  modification 
recommendations.  Recommended  changes  were  subsequently 
incorporated  in  the  final  model. 


STRUCTURED  HYPERMEDIA  APPLICATION  DEVELOPMENT  MODEL  (SHADM) ; 


A  Structured  Model  for  Technical  Documentation 
Application  Design 

I .  Introduction 

Effective  and  efficient  utilization  of  resources  is 
arguably  one  of  the  most  significant  components  of 
operational  management.  During  periods  of  seemingly 
unlimited  budgetary  allocations  there  is  a  tendency  to  not 
always  recognize  this  as  an  important  function  and  tight 
cost  management  becomes  indulgent.  As  finance  is  not 
considered  a  critical  resource,  managers  may  not  be  as 
rigorous  or  objective  in  their  evaluation  of  project 
proposals  and  their  predicted  efficiency  returns.  This  has 
resulted  in  acceptance  of  projects  that  are  not  essential 
and  congruent  with  organizational  goals.  Given  the  absence 
of  restrictive  finance  budgets  the  urgent  need  for  better 
utilization  of  assets  and  constant  improvement  is  relieved. 
External  and  internal  environment  scanning  for  performance 
and  efficiency  enhancing  technologies  is  not  performed. 

Given  the  present  financial  environment  and  facing  the 
potential  for  additional  reductions  in  defence  budgetary 
allocations,  the  Royal  Australian  Air  Force  (RAAF)  must 
continue  to  focus  on  stringent  cost  control.  The  onus  is  on 
all  levels  of  RAAF  management,  now  more  than  ever,  to  ensure 
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that  only  essential  and  cost  effective  practices  are 
introduced  and  maintained  to  support  operational  capability. 

One  area  that  has  significant  potential  for  greater 
efficiencies  in  cost  control  is  that  of  logistics  support, 
particularly  the  management  of  publications.  To  support  its 
operations,  the  RAAF  presently  produces  and  manages  in 
excess  of  17,000  technical  and  administrative  publications 
at  an  annual  cost  of  approximately  AD$12  million.  These  are 
distributed  on  a  national  and  international  basis. 

RAAF  publications  are  Australian  military  handbooks 
that  contain  the  rules,  regulations,  technical  and 
administrative  knowledge  used  by  RAAF  and  Department  of 
Defence 'personnel  in  the  daily  operation  of  the  RAAF.  The 
publications  are  produced  and  distributed  in  semi-pliant 
plastic  four  post  binders  and  the  contents  are  printed  on 
each  side  of  A4  size  paper.  Each  binder  has  an  external 
identification  card  affixed  to  the  front  detailing  the 
subject  information  contained  in  it. 

Publications  are  divided  into  groups  following  their 
general  area  of  specialization;  the  groups  are: 

a.  Education  and  Training 

b.  Supply 

c.  Defence  and  Security 

d.  Administration 

e.  Medical 

f.  Technical 

g.  Air  and  Operations  (23:1-2) 
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Each  specialist  area  is  further  divided  into  sequentially 
numbered  topics  containing  detailed  information  and 
instructions  pertaining  to  that  area. 

RAAF  administrative  and  regulatory  publications  derive 
their  content  from  the: 

a.  Australian  Minister  for  Defence,  in  the  form  of 
instructions  and  directives 

b.  Chief  of  the  Air  Staff,  in  the  form  of 
administrative  directions  governing  the  day  to  day 
operation  of  the  RAAF 

c.  Australian  Government  Department  of  Finance 
Regulations 

Technical  and  maintenance  publications  are  sourced  from: 

a.  Manufacturers  of  weapons  platforms  and  supporting 
technical  equipment,  in  the  form  of  technical 
specifications  and  maintenance  documents 

b.  Experienced  in-house  specialists 

An  integral  element  of  the  production  cost  of  technical 
and  maintenance  publications  is  the  development  of  in-house 
corporate  kr.owledge  and  the  training  and  development  of 
experts  to  carry  out  this  task.  Presently  there  are  in 
excess  of  268  publication  sponsors,  or  "experts",  whose 
primary  roles  are  the  evaluation  and  conversion  of  acquired 
technical  information  into  publication  format. 

The  conversion  of  technical  information  takes  on  a 
number  of  forms;  rewriting  an  acquired  text  in  toto  so  that 
it  relates  to  specifically  RAAF  configured  equipment; 
disregarding  acquired  data  and  writing  a  publication  from 
scratch  based  on  experiential  knowledge;  transferring  an 
acquired  publication  into  an  RAAF  publication  binder  and 
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making  service  unique  changes  through  the  insertion  of 
additional  pages  (usually  pink  in  color  to  allow  for  easier 
differentiation) . 

The  end  result  of  this  approach  to  information 
acquisition  and  dissemination  is  a  labor  intensive  system 
and  an  end  product  that  is  far  from  perfect.  The  user  is 
faced  with  a  large  number  of  technical  and  maintenance 
publications  that  are  presented  in  varying  formats  thus 
making  information  difficult  and  time  consuming  to  find. 

Add  to  this  the  requirement  for  the  user  to  incorporate  a 
large  and  continuous  volume  of  amendments  and  publication 
updates,  to  both  technical  and  administrative  publications, 
and  a  significant  amount  of  expensive  man-time  is  lost. 
Inefficiencies  of  this  magnitude  can  no  longer  be  tolerated 
in  times  of  reduced  manpower  and  funding. 

Problem  Statement 

The  field  of  technical  publications  is  one  where 
information  acquisition,  delivery  and  maintenance  costs  are 
relatively  high  and  application  of  technology  in  information 
delivery  systems  is  low.  The  book  has  been  an  efficient  and 
well  accepted  delivery  vehicle  and  studies  in  the  field  of 
artificial  intelligence,  particularly  expert  systems  and 
hypertext,  indicate  that  a  suitable  replacement  for  the  book 
may  have  been  found  in  Hypermedia  applications  (22:65) 

(26:63)  (20:116)  (17:91). 
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The  objective  of  this  study  is  to  determine  how  an  RAAF 
technical  maintenance  publication  can  be  transformed  into  a 
task  oriented  Hypermedia  application,  and  to  develop  a  model 
that  can  be  used  for  similar  future  transformations.  This 
will  allow  the  RAAF  to  reduce  costs  associated  with 
technical  maintenance  publication: 

a.  acquisition 

b.  conversion 

c.  distribution 

d.  storage 

e.  updating 

It  will  provide  maintenance  personnel  with  a  standard 
publication  format  and  layout  as* well  as  allow  more 
efficient  access  to  information.  Resolution  of  the  specific 
research  problem  will  be  determined  by  answering  the 
following  objective  statements: 

What  is  a  RAAF  technical  maintenance  publication? 

What  is  Hypermedia? 

What  is  a  task  oriented  applies. tion? 

How  can  a  maintenance  publication  be  transformed  into  a 

task  oriented  Hypermedia  application? 

What  is  an  appropriate  model  structure  for  future  use? 

What  criteria  needs  to  be  considered  in  the  model 

design? 

What  general  cost  benefits  can  be  realized  from  this 

application? 
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Scope 


The  objective  of  this  research  is  to  determine  how  an 
RAAF  technical  maintenance  publication  can  be  converted  into 
a  Hypermedia  application.  To  design  and  test  a  Hypermedia 
model  that  will  guide  publication  managers,  or  sponsors,  in 
the  transformation  of  manufacturers  technical  specifications 
and  maintenance  documentation  into  RAAF  specific  task 
oriented  publications  within  a  Hypermedia  environment. 

Technical  maintenance  documentation  evaluation  and 
Hypermedia  model  development  will  be  based  on  USAF  Job 
Guides  prepared  for  the  F-15E  aircraft,  due  to  the 
unavailability  of  RAAF  technical  publications.  The  highly 
structured  nature  of  the  job  guides,  their  availability  and 
the  trend  towards  simplified  text  and  graphics  presentation 
make  them  ideal  subjects  for  model  construction.  The 
differences  between  the  two  services  are  minimal  in  the  area 
of  technical  maintenance,  particularly  as  similar  aircraft 
types  are  operated  and  technical  publications  are  sourced 
from  original  equipment  manufacturers  (OEM) ,  therefore  the 
applicability  of  research  findings  and  final  model  to  the 
RAAF  environment  will  be  assured. 

Resolution  of  research  question  8  will  be  limited  to 
identification  of  general  cost  areas  and  potential  savings 
realized  through  elimination  of  redundant  processes.  The 
limitations  imposed  by  the  academic  requirements  of  AFIT  and 
the  amount  of  time  available  for  this  study  preclude  more 
detailed  cost/benefit  analysis. 
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II. 


Literature  Review 


RAAF  Technical  Maintenance  Publications  Deficiencies 

As  indicated  in  the  introduction,  the  present  RAAF 
technical  maintenance  publication  system  is  producing 
publications  in  a  variety  of  formats.  Inherent  with  this 
approach  are  inefficiencies  in  conversion,  production  and 
use.  There  is  no  consistency  in  conversion,  not  all 
publications  are  fully  reworked  into  RAAF  format  and  no 
formal  guidelines  exist  to  help  the  individual  publication 
managers  in  identifying  publications  suitable  for 
conversion,  although  there  are  extensive  regulations 
detailing  the  actual  acquisition  and  management  process 
(23) .  The  decision  to  convert  is  based  on  a  heuristic 
process  exercised  by  the  relevant  publication  manager.  This 
lack  of  consistency  results  in  maintenance  documentation  in 
varying  formats  being  placed  in  the  unit  workshops  which 
place  an  added  burden  on  the  users  requiring  them  to  be 
familiar  with  each  layout  and  indexing  style. 

Two  investigations  were  conducted  into  the  management 
of  publication  production  in  the  RAAF.  The  first  was  in 
1978,  its  goal  being  to  evaluate  the  replacement  of  printed 
publication  medium  with  microfiche  (24:1).  The  second,  in 
1981,  at  the  request  of  the  Senior  Administrative  Staff 
Officer,  Headquarters  Support  Command,  was  tasked  to 
specifically  study  increasing  expenses  and  reducing 
efficiency  in  publications  production.  While  this  study 
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concentrated  on  the  physical  printing  aspects  of  the 
production  cycle  (6)  both  reports  referenced  the  problems 
outlined  above.  The  recommendations  of  the  1978  report  were 
to  replace  technical  maintenance  paper  publications  with 
microfiche  as  it  was  considered  a  superior  medium  that 
removed  the  excessive  and  convoluted  production  process  and 
eliminated  the  requirement  for  field  users  to  continually 
update  publications.  The  1981  report  recommended 
replacement  of  deficient  and  redundant  printing  and 
production  machinery.  Of  the  two,  only  the  1981 
recommendations  were  acted  upon  and  despite  the  earlier 
findings,  no  further  action  has  been  taken  to  address  this 
aspect  of  technical  maintenance  publication  management. 

Expert  Systems 

A  good  starting  point  for  a  definition  and 
understanding  of  what  an  Expert  System  is  is  to  define 
Artificial  Intelligence,  "The  study  of  mental  faculties 
through  the  use  of  computational  models  by  attempting  to 
duplicate  (or  simulate)  the  mental  faculties  and  thought 
processes  of  an  individual,  as  in  speech  and  vision"  (19:1); 
and  an  Expert  System  is  "a  computer  based  system  that  uses 
knowledge,  facts,  and  reasoning  techniques  to  solve  problems 
that  normally  require  the  abilities  of  human  experts" 

(19:14) . 

Expert  Systems  are  designed  to  replace  "experts"  in 
decision  making  processes  by  acting  as  repositories  of 
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knowledge.  Researchers  are  divided  in  opinion  on  the  extent 
to  which  Artificial  Intelligence  and  Expert  Systems  can  be 
applied  or  are  successful  in  replacing  the  human  element  in 
decision  making,  given  that  studies  have  revealed  that  they 
are  suitable  applications  where  the  interest  domains  are 
very  narrow  or  highly  bounded  (7:4-7).  Given  that  a 
significant  proportion  of  human  decision  making  is  unbounded 
(7:4-7)  and  involves  the  application  of  heuristics  or 
intuitive  knowledge,  traditional  researchers  are  skeptical 
regarding  the  level  of  application  and  see  AI  and  expert 
systems  in  a  supportive  role  similar  to  decision  support 
systems  (DSS)  (13:333).  Henderson  examines  both 
applications  and  appropriately  and  succinctly  concludes  that 
"DSS  a; nd  expert  systems  technologies  can  offer  opportunities 
to  enable,  and  perhaps  provide,  a  catalyst  for  major 
organizational  transformations”  (13:345).  Introduction  of 
these  technologies  in  the  RAAF  publication  system  would 
result  in  the  need  for  a  significant  review  of  the  current 
publication  management  structure.  The  ramifications  in 
relation  to  organizational  transformation  would  extend  far 
beyond  the  immediate  publications  environment  and  would 
force  the  review  of  manning  and  career  structures  for  a 
number  of  officer  and  airman  job  streams  as  the  need  to 
employ  people  in  the  production  and  management  spheres 
becomes  redundant. 
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Hypermedia 


To  better  understand  Hypermedia  and  its  role  in  on-line 
technical  maintenance  documentation  a  clear  definition  of 
what  it  is  and  where  it  fits  into  the  overall  hierarchy  of 
data  processing  is  essential.  Although  the  terms  Hypermedia 
and  Hypertext  are  often  used  synonymously,  Hypermedia  was 
spawned  from  Hypertext;  Hypermedia  incorporates  basic 
elements  of  Expert  Systems,  Hypertext  and  Artificial 
Intelligence  technology  (16:4). 

In  general  terms,  "Hypertext  may  be  viewed  as 
multidimensional  text  space  constructed  as  associative  links 
within  and  between  documents"  (12:45),  or  "a  combination  of 
natural  language  text  with  the  computer’s  capacity  for 
interactive  branching,  or  dynamic  display... of  a  nonlinear 
text. . .  which  cannot  be  printed  conveniently  on  a 
conventional  page"  (10:17).  Multidimensionalism  is 
achieved  through  the  use  of  "nodes", and  "links"  and  is  best 
represented  graphically,  see  Figure  1.  The  nodes  are  the 
files  or  pages  of  text  while  the  links  are  the  joining 
points  bet’.een  t'  e  files.  The  links  between  the  nodes 
provide  the  nonlinearity  of  Hypertext  and  can  be  compared  to 
footnotes  or  bibliographic  references  (10:33)  where  the 
reader  can  access  them  and  "pursue  ideas  suggested  by  one 
level  of  a  document  to  another  level  or  document"  (12:45) 
without  physically  leaving  the  text  under  study.  Put  more 
succinctly,  nodes  can  be  described  as  single  "concepts"  or 
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"ideas”  while  links  are  the  relationships  or 
"interdependencies"  between  them  (10:37). 


Hypertext  database 


Figure  1:  Hyp^^rt-^xt  Documents  Showing  Nodes  and  Links  (in:i8i 


Hypermedia  encompasses  the  power  of  Hypertext  as  an 
information  database  retrieval  system  (10:33)  and  combines 
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it  with  additional  information  storage  and  presentation 
media  such  as  graphics,  audio,  animation  and  video;  in 
combination  with  heuristic  based  expert  system  applications, 
this  produces  an  extremely  powerful  and  flexible  knowledge- 
based  information  storage  and  retrieval  system.  User  query 
responses  can  therefore  be  presented  in  a  number  of  ways 
dependent  upon  the  type  of  query  made  and  is  described  by 
Franklin  as: 

A  hypermedia  computer  system  integrates  media  to 
facilitate  the  associative  retrieval,  manipulation,  and 
storage  of  information.  The  associative  links  among 
pieces  of  information  allow  the  computer  to  mimic  human 
memory.  Words,  pictures,  and  sounds  can  be  linked  to 
represent  an  idea  or  to  aid  in  problem  solving.  While 
hypermedia  cannot  fully  represent  realia,  other  media 
such  as  music,  video,  text  and  graphics  are  ideal. 
(12:44) 


Application  of  Hypermedia 

The  particular  area  of  interest  for  this  study  is  in 
the  field  of  "macro  literary  systems"  (10:20).  This  is  "the 
study  of  technologies  to  support  large  on-line  libraries  in 
which  interdocument  links  are  machine-supported" ( 10 : 20) ;  in 
particular  their  applicability  to  a  military  environment 
supporting  highly  technical  repair  and  maintenance 
operations  specifically  at  the  task  achievement  level.  The 
ability  of  Hypertext  systems  to  operate  successfully  in  this 
environment  was  demonstrated  by  the  application  of  the  ZOG 
hypertext  system  on  the  USS  Carl  Vinson  (5:2).  Although  ZOG 
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was  a  prototype  it  was  successfully  used  to  support  four 
applications; 

”1.  On-line  policy  manual  (Ship's  Organisation  and 
Regulation  Manual) 

2.  Interactive  task  management  system  (for  analyzing 
and  tracking  complex  tasks) 

3.  On-line  maintenance  manual  with  interface  to 
videodisc  (for  weapons  elevators) 

4.  Interact  to  the  Air  Plan  expert  system"  (5:2). 

The  lessons  learned  from  the  ZOG  experiments  were  applied  to 
a  commercial  version  called  KMS  (Knowledge  Management 
System)  (5:2)  further  establishing  the  suitability  of 
hypertext  in  military  environments. 

The  military  application  of  hypertext  was  developed 
further,  although  on  a  limited  basis  by  Adam  Levitt  in  his 
1989  thesis  (17)  where  he  modelled  a  shipboard  messaging 
system  using  hypertext.  He  concluded  that,  as  a  technology, 
hypertext  offered  exciting  and  promising  alternatives  to  the 
current  "paper"  system  (17:90),  although  he  did  identify  the 
need  to  further  address  principles  and  design  issues  that 
are  consistent  with  present  military  policies  (17:91). 

USAF  Technical  Maintenance  Publications 

The  USAF  Technical  Maintenance  Publications  produced 
for  the  F-15E  aircraft  were  published  using  the  Maintenance 
Integrated  Data  Access  System  (MIDAS)  developed  by  McDonnell 
Aircraft  Company  in  response  to  the  Technical  Order  (TO) 
specification  requirement  detailed  in  MIL-M-83495  (19).  The 
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revised  publication  structure  provides  the  following 
documentation : 

a.  General  Vehicle  Manual  (GV) 

b.  General  System  Manual  (GS) 

c.  Job  Guide  (JG) 

d.  Fault  Reporting  Manual  (FR) 

e.  Fault  Isolation  Manual  (FI) 

f.  Schematic  Diagram  Manual  (SD) 

g.  Wiring  Data  Manual  (WD)  (21:2-3) 

General  Vehicle  Manual.  The  object  of  the  GV  is  to 
provide  "an  aircraft  description  and  maintenance  orientation 
and  is  designed  to  give  the  technician  a  broad  view..",  of 
the  aircraft  vehicle  and  "..the  maintenance  manuals  needed 
to  support  it"  (21:3).  This  publication  contains  the 
following  information: 

a.  General  maintenance  information 

b.  Description  of  the  maintenance  manual  set 

c.  Short  systems  description 

d.  Details  of  time  compliance  technical  orders 

e.  General  maintenance  procedures 

f.  Danger  areas,  precautions  and  precautionary 
procedures 

g.  Vehicle  measurements 

h.  Information  relevant  to  vehicle  lifting,  jacking, 
levelling  and  towing 

i.  Consolidated  list  of  supplies,  test  equipment  and 
special  tools 
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j.  Abbreviations  and  symbols  used  (4:1-1) 

General  System  Manuals.  The  GS  manuals  address  the 
major  aircraft  systems  and  provide  information  on  the 
following  areas: 

a.  System,  subsystem  and  sub-subsystem  descriptions 

b.  Principles  of  operation 

c.  List  of  supplies,  test  equipment  and  special  tools 

d.  List  of  Line  Replaceable  Units  (LP.U's) 

e.  List  of  abbreviations  and  symbols  (4:1-1) 

Job  Guides.  JG's  are  the  task  oriented  element  of  the 
publication  system  and  provide  the  step  by  step  detail 
required  by  the  user  to  perform  operations  on  vehicle  sub 
and  sub-subsystems.  Specifically  they  address: 

a.  The  functions  of  each  LRU 

b.  The  performance  tasks  associated  with  each  LRU 
function  including: 

1.  Removal 

2.  Installation 

3.  Inspection 

4.  Cleaning 

5.  Operational  Checkout 

6.  Adjustment 

7.  Calibration  (4:1-1) 

c.  Maintenance  practices  and  functions  covering: 

1.  Applicability 

2.  Required  conditions 

3 .  Recommended  personnel  to  perform  task 
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4 .  Support  equipment  required 

5.  Support  data 

6.  Applicable  load  specifications 

7.  Supplies  required 

8.  Personal  safety  equipment  needed 

9.  Safety  condition 

10.  Checklist  requirements 

Fault  Reporting  Manual.  The  Fault  Reporting  Manual 
provides  the  vehicle  for  aircrew  and  maintenance  staff  to 
develop  "fault  codes"  (21:3)  that  identify  aircraft 
malfunctions  and  eventually  allow  for  identification  of 
specific  faulty  LRU's.  The  publication  has  three 
significant  components: 

a.  "Alphabetical  index  with  precise  statements  of 
possible  malfunctions  for  every  major  aircraft 
system  and  subsystem"  (21:3). 

b.  "A  built-in  test  (BIT)  fault  indications  index 
keyed  to  the  avionics  status  panel  fault  indicator 
numbers"  (21:3)  . 

c.  "A  pictorial  index  which  is  an  illustrated  view  of 
the  cockpit  with  all  gages  and  indicators  fully 
detailed"  (21:4)  . 

Fault  Isolation  Manuals.  These  manuals  are  used  in 
conjunction  with  the  Fault  Reporting  Manuals  and  are  used 
after  the  specific  fault  codes  have  been  identified  during 
the  post  flight  debriefing.  They  provide  the  technicians 
with  a  clear  logic  tree  approach  to  the  aircraft  systems  and 
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all  possible  faults  that  can  be  experienced  with  each  LRU. 
Using  the  codes  and  the  "yes/no  logic  approach"  (21:4)  the 
technician  is  able  to  quickly  locate  and  repair  the  reported 
fault. 

Wiring  Data  Manuals.  These  publications  provide 
diagrammatic  format  explanations  of  the  aircrafts' 
electrical  wiring  layouts. 

Schematics  Diagram  Manuals.  The  SD  manuals  provide 
"the  schematic  diagrams  required  for  troubleshooting 
malfunctions"  (4:1-4).  They  combine  electrical  and 
hydraulic  schematics  in  one  manual  for  non-complex  systems, 
however,  they  are  separated  for  the  -  e  complex  units 
(4:14)  . 

Technical  Order  Numbering.  The  TO  numbering 
requirements  and  structure  are  detailed  in  MIL-M-83495  and 
are  designed  to  enable  the  user  to  identify  "which  model 
aircraft  the  TO  supports;  whether  the  TO  supports  a  sub  or 
sub-subsystem;  the  volume  number  of  the  TO;  and  what  type  of 
information  the  TO  provides"  (4:1-5). 
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Cost  Comparison 


Present  System.  Under  the  present  system  of 
publication  acquisition,  development  and  production  in  the 
RAAF  the  following  significant  cost  areas  can  be  identified: 

a.  Acquisition  -  the  cost  of  purchasing  technical 
maintenance  documentation  and  update  service.  If  the 
publications  are  prepared  by  the  OEM  in  user  format 
then  the  cost  of  production  will  be  included  in  the 
purchase  price. 

b.  Evaluation  and  Conversion  -  this  is  where  the 
publication  sponsor  or  manager  evaluates  the 
documentation  and  converts  it  to  RAAF  specific 
terminology  and  maintenance  procedures. 

c.  Production  -  the  input,  manipulation,  formatting 
and  production  of  paper  hardcopy  publications  for 
distribution  to  users.  This  involves  both  Australian 
Government  printing  facilities  as  well  as  civilian 
printing  contractors. 

d.  Storage  and  Distribution  -  identifying  and 
physically  distributing  publications  to  user  units. 

e.  Unit  distribution  and  maintenance  -  includes  the 
receipt,  recording  and  distributing  at  user  level  of 
publications.  This  area  also  includes  the  cost  of 
maintaining  the  documentation  through  periodical 
updates  and  amendments. 
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Hypermedia  System.  Under  a  Hypermedia  technical 
maintenance  documentation  system  the  following  major  cost 
areas  are  identified: 

a.  Data  acquisition  -  under  the  CALS  Information  Data 
Exchange  System  weapons  platforms  publication  data  will 
be  able  to  be  transferred  electronically  thus 
formatting  can  be  conducted  at  either  contractor  or 
System  Project  Office  (SPO)  leyel.  If  the  contractor 
completes  this  process,  giyen  that  formatting  was 
previously  done  for  hardcopy,  the  cost  of  paper  copy 
production  would  be  negated.  If  the  publication 
manager  at  the  SPO  is  required  to  format  the  document 
then  this  would  be  the  only  production  cost  involved  in 
the  conversion. 

b.  Distribution  -  with  the  technical  data  in 
electronic  format,  distribution  to  end  users  can  be 
accomplished  through  either  existing  computer 
communications  networks  or  by  transferring  the  data  to 
magnetic  or  optical  storage  media  and  uti^-izing  present 
mail  channels  for  distribution.  This  would  produce  a 
significant  cost  reduction  in  carrying  charges  for 
hardcopy  publications.  Further,  large  stockpiles  of 
surplus  publication  holdings  held  to  meet  future 
demands  for  replacement  books  due  to  normal  and 
abnormal  wear  and  tear  attrition,  representing  a  major 
sunk  cost,  would  not  be  required. 
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c.  Maintenance  -  as  updating  of  publications  would  be 
conducted  at  the  SPO  prior  to  transmission  to  users,  no 
costs  would  be  incurred  at  unit  level  in  maintaining 
publications.  Additionally,  publication  integrity  can 
then  be  assured  as  the  potential  for  incorrect 
amendment  by  users  is  negated  further  reducing  the 
cause  of  incidental  on  cost  through  incorrect 
maintenance  procedures. 

Conclusion 

This  chapter  detailed  the  background  to  the  problem 
under  study,  defined  the  research  question  and  laid  out  the 
investigative  questions  that  support  the  research  problem. 
The  scope  of  the  application  was  also  defined,  acknowledging 
the  limitations  placed  on  the  study  by  academic  and  time 
restrictions.  Topical  literature  was  reviewed  relative  to 
expert  systems,  hypertext  and  hypermedia.  Finally,  a 
generalized  cost/benefit  comparison  was  conducted  between 
the  current  RAAF  publication  system  and  the  Hypermedia 
system  under  research  to  identify  potential  cost  savings. 
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Ill .  Methodology 


Literature 

As  a  research  methodology,  prototyping  has  significant 
benefits  for  the  researcher: 

a.  It  provides  a  live  working  application 

b.  Enables  the  quick  development  of  a  working 
application 

c.  Provides  for  the  testing  of  assumptions  made  by 
the  analyst  and  user  relative  to  system  features 

d.  Allows  for  iterative  development 

e.  Is  cost  effective  (19:15) 

Figure  2  details  a  flowchart  of  the  basic  steps  in 
prototyping. 

The  basic  concept  behind  prototyping  is  to  quickly 
produce  a  working  model  of  the  system  being  prototyped,  one 
that  has  the  look  and  feel  of  the  desired  system,  and  for 
the  developer  to  work  with  the  user  to  refine  the  model  to 
meet  the  user  requirement  (1:274). 

Proposed  System 

In  this  study  the  following  steps  will  be  utilized  in 
developing  the  technical  maintenance  documentation  model: 
a.  Two  current  USAF  technical  maintenance 
publications  for  F15E  aircraft  maintenance  tasks  will 
be  selected  and  analyzed  to  determine  basic  structure 
and  data  segmentation.  In  particular  commonality  in 
structure,  task  division  and  relationships  between  data 
segments  will  be  identified  to  facilitate  the 
development  of  a  transformation  model  to  convert 
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Figure  2:  Steps  in  Prototype  Development  Method  (25:221', 


existing  and  future  technical  maintenance  information 
to  Hypermedia  format. 

b.  A  prototype  model  of  a  Job  Guide  structure  will  be 
developed  using  Hyperwriter  development  software.  The 
prototype  will  be  discussed  and  analyzed  with  USAF 
publication  managers,  engineers,  senior  aircraft 
technicians,  an  RAF  engineer  and  an  RAAF  publication 
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manager,  to  be  selected  from  the  F-15E,  C-17  SPO's, 

AFIT  and  the  RAF  Liaison  office.  The  prototype  will  be 
further  developed  as  appropriate  to  meet  the  users' 
requirements . 

c.  In  concert  with  the  selected  evaluation  personnel, 
a  specific  job  guide  task  will  be  selected  for 
transformation  to  Hypermedia  to  test  the 
appropriateness  of  the  prototype  model. 

d.  The  Hypermedia  model  will  be  evaluated  by  the 
selected  personnel  through  their  testing  of  the  model 
and  success  or  failure  will  be  judged  by  questionnaire 
responses . 


Hypermedia  Design  Considerations 

While  the  relevance  and  applicability  of  Hypermedia  in 
the  military  environment  has  been  determined,  its 
suitability  as  a  replacement  for  technical  maintenance 
documentation  and  structure  design  considerations  must  be 
identified  prior  to  any  acceptance  of  application.  As  an 
absolute  essential: 

Any  electronic  information  delivery  system  must  be  able 
— at  a  very  minimum —  to  duplicate  the  capabilities 
provided  by  the  combination  of  an  experienced  reader 
and  a  well-designed  text.  Anything  less  degrades  the 
system:  leaving,  at  best,  an  electronic  page-turner; 

at  worst,  even  less  than  the  paper  version. 

Furthermore,  in  order  to;  justify  abandoning  the 
conventional  method  and  medium,  an  electronic  system 
should  offer  improvements,  such  as  increased 
flexibility,  reduction  in  storage,  and  convenient 
document  development  and  maintenance.  (9:99) 
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Document  Design  Strategies 


In  Hypertext  document  structuring,  information  is 
broken  into  logical  "chunks"  and  then  joined  by  "links". 
While  document  structuring  strategies  range  from  the  simplex 
to  the  complex,  there  is  concurrence  on  the  need  to  provide 
the  user  with  an  easily  navigable  document  to  avoid 
confusion  and  disorientation  within  the  document  while 
following  information  paths.  Van  Dyke  Parunak  offers  a 
number  of  potential  designs  ranging  from  the  hierarchical 
array  to  the  highly  complicated  Hypertorus  (28:46).  Van 
Dyke  Parunak  provides  a  good  insight  into  design  strategies 
and  document  topologies  yet  makes  no  mention  of  the  needs  of 
the  user  and  the  users  familiarity  with  the  content  of  the 
document. 

The  topic  of  user  interface  and  document  design  raises 
a  number  of  questions  on  the  non-linearity  of  specific 
design  strategies.  Present  thinking  on  interaction 
interface  designs  present  many  strategies  for  coping  with 
the  problem  of  different  naming  or  referencing  styles 
between  the  user  and  the  developer.  The  major  approach  used 
in  document  design  is  a  writer  directed  movement  through  the 
hypertext  document.  This  appears  contrary  to  the  concept  of 
non-linear  reading  in  that  it  removes  the  ability  of  the 
reader  to  branch  when  and  where  he  feels  relevant  to  his 
information  needs  or  interest  as  the  information  presented 
is  controlled  by  the  developer.  Also,  for  the  user  to 
successfully  use  this  approach  a  common  language  or 
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terminology  is  essential  between  the  two.  If  this  can  be 
achieved  the  use  of  implicit  links  based  on  keywords 
supplied  by  the  user  as  a  "guess”  (14:53)  can  minimize  the 
possibility  of  the  user  becoming  lost  or  disorientated 
within  the  document  structure.  Inherent  in  this  approach  is 
the  problem  of  large  overhead  and  confusion  for  the  user 
through  excessive  numbers  of  successful  "hits"  on 
keyword/link  finds.  This  slows  the  system  down  as  the  user 
is  then  required  to  assess  each  success  for  its  relevance, 
also  having  a  large  number  of  successes  also  places  greater 
data  management  overhead  on  the  system  with  increased  demand 
on  memory  and  disk  access. 

A  writer  directed  approach  to  information  access 
generally  implies  the  utilization  of  a  hierarchical  document 
design  (14:52).  As  stated  previously,  the  writer  directed 
approach  presupposes  a  number  of  things;  firstly,  that  the 
writer  knows  what  information  is  relevant  to  the  reader  and 
in  what  order  he/she  will  want  to  access  it;  secondly,  that 
this  type  of  approach  reverts  the  document  to  a  quasi-1 inear 
format,  losing  the  benefits  attributable  to  the  non¬ 
linearity  of  a  true  Hypertext  document. 

Information  relevancy  is  a  critical  aspect  of  good 
Hypertext  document  design  and  in  an  unstructured  information 
delivery  system  significant  difficulty  can  be  encountered  in 
determining  the  right  structure  and  order.  A  way  of 
reducing  this  problematic  aspect  is  to  conduct  rigorous 
analysis  of  users  and  user  needs,  in  particular  how  and  when 
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the  user  requires  or  will  query  the  system.  Under  these 
conditions  it  is  not  feasible  to  produce  one  document 
structure  that  meets  all  users  requirements  (14:48).  One 
approach  to  overcoming  this  is  to  structure  the  document 
relative  to  the  experience  level  of  the  user  (14:49).  A 
"novice"  or  beginner  would  require  more  information  of  an 
explanatory  or  background  nature  than  would  an 
"intermediate"  or  "expert"  level  user  (14:49).  Therefore, 
an  alternate  design  strategy  would  take  this  into 
consideration  and  produce  a  document  that  would  cater  not 
only  to  the  needs  of  the  user  but  the  users  experience 
level . 

The  problems  identified  thus  far  in  relation  to 
Hypertext  document  structure  lend  significant  support  to  the 
adoption  of  a  task-oriented  structure.  This  negates  the 
requirement  for  the  designer  to  pre-guess  the  user's 
information  needs,  particularly  in  a  task  that  is  highly 
structured  and  procedural  as  in  aircraft  maintenance 
publications.  However,  this  does  not  remove  the  need  for 
the  background  or  explanatory  information  to  be  available  to 
the  user  but  affords  the  designer  the  opportunity  to  provide 
them  as  a  choice  option  rather  than  as  a  rigid  part  of  the 
pathway  through  the  procedural  elements  of  the  document. 

Human  Aspects 

In  managing  organizational  change,  particularly  with  the 
introduction  of  new  technology,  the  human  aspects  must  be 
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considered  among  the  priorities  of  change  managers. 
Inappropriate  approaches  to  this  element  can  result  in  both 
overt  and  covert  resistance  to  the  systems  being  installed. 
This  is  just  as  true  in  a  military  environment  as  in  a 
civilian  one  and  was  evidenced  during  the  writer's 
involvement  in  the  development  of  a  computerized  database 
system  at  RAAF  Publications,  Melbourne.  There,  personnel 
unfamiliar  with  the  new  technology,  felt  threatened  and  the 
introduction  phase  of  the  project  was  extremely  difficult. 
These  problems  are  supported  by  Bjorn-Andersen  (8)  who 
addresses  the  "human"  factors  in  computer  systems 
development  and  -  .ication  focusing  in  particular  on  the 
propensity  of  stems  developers  to  modify  applications  with 
"human  wea::nesses"  to  make  them  more  acceptable  to  the  end- 
user  (8:386).  Of  significance  is  his  development  of  the 
"Scandinavian  Model"  of  system  development  which  is  heavily 
user  oriented  (8:389).  The  specific  points  raised  by  Bjorn- 
Andersen  are  supported  by  Hirschheim  and  Newman  (15)  who 
present  a  case  study  of  an  introduction  of  technological 
change  (15:405).  From  tne  reports  reviewed  three  common 
areas  of  significance  relative  to  the  development  of  this 
Hypermedia  application  are  identified: 

a.  Ensure  the  system  design  is  appropriate  (15:400) 

b.  Allow  the  user  to  become  involved  in  the  system 
design  (8:389) 

c.  Provide  personnel  with  complete  and  comprehensive 
training  (8:390) 
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Hirschheim  and  Newman  conclude  "The  development  and 
implementation  of  computer-based  information  systems  is  a 
type  of  major  organizational  change.  Only  those  development 
strategies  which  view  such  change  in  terms  of  social  and 
political  processes  are  likely  to  prove  satisfactory" 
(15:406) . 

System  Design 

As  detailed  in  the  objective  of  this  study,  the  USAF 
Job  Guide  was  used  as  the  basis  for  the  design  of  the 
Hypermedia  development  model  because  of  its  stylized 
structure  allowing  ease  of  conversion  to  hypertext  nodes. 
This  was  not  always  the  case.  During  the  period  1962  to 
1975,  because  of  its  complex  structure  and  its  inappropriate 
style  for  maintenance  procedures,  the  format  of  the  guide 
was  modified  from  a  "full  size"  structure  to  its  present 
detailed  step-by-step  procedural  presentation  including 
explicit  illustrations  (26:5).  This  particular  structure 
was  validated  during  a  1975  survey  by  Air  Force  Human 
Resources  Laboratory  (AFHRL)  conducted  with  maintenance 
technicians  actively  using  the  guides  for  day  to  day 
maintenance  tasks  (26:5). 

Users  of  on-line  data  retrieval  systems  require  a 
number  of  things  from  the  system  to  ensure  'usability'; 
these  include: 

a.  Simplicity  of  layout  and  information  structure 

b.  Clarity  of  information 
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c.  Understandability  (27:320) 

In  addition  a  good  program  design  should  "provide 
backtracking,  Boolean  search  capabilities,  and  other  methods 
for  easy  navigation"  (12:46).  From  a  structural  viewpoint 
these  requirements  are  met  by  the  current  job  guide 
structure,  as  validated  by  the  1975  AFHRL  survey,  combined 
with  the  introduction  of  simplified  English  and 
illustrations  currently  under  way.  Researchers  concur  that 
an  analyst  requires  a  common  framework  within  which  to 
develop  his  specific  application  (29:30).  This  is 
particularly  so  in  an  environment  where  many  applications 
follow  the  same  format  structure. 

While  the  structure  of  job  guides  lends  itself  to 
standardization,  the  actual  content  of  the  documents  will 
not  and  in  each  case  may  require  individual  analysis. 

Figure  3  shows  a  model  of  this  process,  and  while  the 
structure  half  of  the  model  has  been  taken  care  of  by  the 
adoption  of  the  present  structure,  the  content 
representation  identifies  clearly  what  the  model  under 
development  will  require  from  the  publication  producer, 
whether  this  be  the  OEM  or  the  Service  Publication  Manager. 

The  supportive  role  of  the  other  publications  in  the 
manual  set  is  recognized  and  a  detailed  analysis  was 
conducted  to  identify  the  information  content,  its  structure 
and  inter-relationships.  The  analysis  is  detailed  at 
Appendix  A. 
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Figure  3:  Model  of  the  Representation  Process  (21:254) 

Summary 

This  chapter  identified  current  thinking  and  literature 
in  relation  to  the  prototyping  methodology  and  detailed  the 
proposed  conduct  of  this  research.  Hypermedia  design 
considerations  were  also  discussed  particularly  in  relation 
to  system  design  and  user  interface  aspects. 
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IV.  Model  Design 


Introduction 

The  objective  of  this  chapter  is  to  develop  and  present 
the  SHADM  model  to  be  used  in  converting  technical 
maintenance  documentation  into  hypermedia  format. 

The  following  areas  are  addressed: 

a.  Standards  and  formats 

b.  User  interface  sub-model 

c.  Data  sub-model 

d.  Data  relationship  sub-model 

Standards  and  Formats 

This  section  describes  the  model  standards  and  formats 
that  should  apply  to  a  hypermedia  application  of  technical 
maintenance  documentation.  A  number  of  significant  aspects 
impinging  on  successful  model  design  were  identified  in  the 
literature  review  conducted  in  Chapter  2.  The  object  of  the 
model  design  phase  is  to  maximize  the  beneficial  aspects  of 
these  elements  giving  cognizance  to  the  other  model  elements 
to  ensure  that  one  is  not  maximized  to  the  detriment  of 
another.  A  fine  line  must  be  followed  in  design  standards 
to  obtain  the  optimum  combination  of: 

a.  User  friendliness 

b.  Ease  of  model  use 

c.  Use  of  hardware  capabilities 

d.  Use  of  software  capabilities 
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Model  robustness 


e . 

f.  Compliance  with  existing  information  storage  and 
transfer  standards  applicable  within  the  military 
environment 

In  developing  appropriate  standards  and  formats  the 
following  areas  are  addressed: 

a.  Text 

b.  Graphics 

c.  Menus 

Text .  Text  files  used  in  a  hypermedia  application 
should  conform  to  the  following  standards: 

a.  Storage.  Text  files  used  in  technical  maintenance 
publications  must  comply,  as  a  minimum,  with  the 
CALS  requirements  detailed  in  MIL-STD-1840A.  In 
particular,  text  should  be  stored  in  7  bit  ASCII 
format  (11:11)  and  in  separate  files  by  vehicle 
system.  This  method  of  storage  maximizes  the  use 
of  both  secondary  and  primary  storage  by  not 
creating  excessively  large  files  that  are 
difficult  to  maintain  ind  address  and  not  placing 
excessive  demands  on  the  CPU  and  primary  storage 
through  large  numbers  of  I/O  requests  caused  by 
significant  numbers  of  small  files.  Where  tables 
or  large  data  screens  contain  logically 
segmentable  data  they  should  be  broken  into 
separate  logical  nodes  and  displayed  in  overlay 
windows.  Their  group  heading  should  be  returned 
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in  the  initial  window  and  provide  the  link  anchor 
point.  This  serves  a  twofold  purpose,  first  it 
avoids  the  user  having  to  scroll  through  large 
screens  of  data  to  locate  a  specific  element. 
Secondly,  the  individual  nodes  provide  greater 
linking  flexibility  to  other  nodes  because  of 
their  greater  content  specificity. 

b.  Presentation.  Data  should  be  displayed  in  an 
article  metaphor  presentation  style  (16:5)  thus 
not  limiting  the  display  capability  to  individual 
25  line  screens.  Article  metaphor  systems  allow 
greater  than  25  lines  and  provide  the  user  with 
the  capability  to  scroll  in  any  direction  through 
the  information.  Where  text  extends  for  greater 
than  30  lines  in  an  overlay  display  box,  to  reduce 
the  need  for  extensive  scrolling,  a  link  to  a  new 
display  window  should  be  used  even  through  the 
text  may  be  of  an  amplifying  nature. 

c.  Color.  Text  presentation  should  be  white 
characters  on  a  black  background.  Cautions  and 
Warnings  text  should  be  displayed  as  red 
characters  on  a  white  background  to  significantly 
differentiate  them  from  normal  text  and  to 
highlight  the  urgent  nature  of  the  information. 

Use  of  red  should  be  limited  solely  to  Warnings 
and  Cautions,  however,  while  other  colors  may  be 
used  sparingly  to  enhance  presentation. 
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consideration  must  be  given  to  the  visual  quality 
and  effect  of  their  use  to  ensure  overall 
presentation  is  not  degraded. 

Graphics .  Graphics  files  used  in  a  hypermedia 
application  should  conform  to  the  following  standards: 

a.  Storage.  Compliance  with  CALS  standards  detailed 
in  MIL-STD's  28000  and  28003  should  be  mandatory 
in  all  graphics  transfer  and  storage.  Common 
graphics  should  be  stored  once  and  reproduced  as 
required  in  a  background  mode.  This  will  reduce 
the  high  secondary  storage  memory  costs  associated 
with  graphic  images. 

b.  Presentation.  All  graphics  should  be  presented  as 
white  lines  on  a  black  background.  Using  the 
background  display  method  for  common  graphics 
detailed  at  sub-para  a.,  different  callouts 
appropriate  to  each  task/ function  should  be 
overlaid  on  the  graphic  to  provide  relationship 
link  points  and  explanatory  text.  Color  should  be 
used  in  graphics  where  it  is  appropriate  and 
enhances  the  ability  of  the  image  to  transfer 
knowledge/ information.  However,  as  with  color 
text  representation,  red  is  restricted  solely  to 
caution  and  warning  messages  and  consideration 
must  be  given  to  the  visual  quality  and  effect. 

The  use  of  additional  color  must  not  degrade  the 
overall  graphic  presentation. 
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c.  Icons .  To  simplify  the  presentation  interface 
icons  should  be  used  to  represent  simple  common 
repetitive  tasks  to  be  performed.  Each  icon 
should  be  stored  once,  used  in  background  mode  for 
display  and  be  globally  defined  to  enable  the 
inexperienced  user  to  ascertain  an  icon 
explanation  by  selecting  it  as  a  link. 

d.  Naming  Standards.  Graphic  names  should  uniquely 
identify  the  task  being  described  and  either  the 
common  name  or  S/S/SN  number  of  the  LRU.  It 
should  be  displayed  as  a  window  title  in  the 
format: 

[TASK]-[LRU  Common  Name]/[LRU  S/S/SN] 
where  use  of  titles  is  dependant  upon  the 
task/ function  being  performed,  e.g.  the  title  of 
the  nose  landing  gear  installation  function  would 
be: 

INSTALLATION  -  NOSE  LANDING  GEAR 
At  the  task  level  of  a  specific  LRU  installation 
the  title  would  be; 

INSTALLATION  -  32-20-10 

This  naming  standard  is  applicable  to  both  textual 
and  graphical  displays.  It  allows  the  user  to 
directly  address  specific  data  windows  dependent 
upon  the  information  sought. 
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Menus ,  The  SHADM  model  supports  three  menu  strata , 
Primary,  Secondary  and  Tertiary.  These  menu  structures 
should  comply  v/ith  the  following  standards: 

a.  Primary  Menus.  Primary  menus  should  allow  the 
user  to  choose  between  graphic  representation 
selection  or  direct  addressing.  Each  selection 
should  lead  the  user  to  a  secondary  menu  which 
offers  a  function  selection. 

b.  Direct  Addressing.  Direct  addressing  allows  the 
user  to  specify  precise  data  windows  as 
destinations  using  window  titles.  The  access  mode 
should  be  interactive  from  a  menu  selection  and 
should  offer  the  user  a  range  of  titles  to  choose 
from. 

c.  Graphic  Representation.  Graphic  representation 
should  display  a  graphic  image  of  the  specific 
vehicle,  at  Primary  menu,  or  system  or  LRU  at  the 
Secondary  menu.  The  image  should  be  divided  into 
link  areas  or  "hot-spots”,  for  vehicle  graphics 
this  should  be  by  system  and  for  system  or 
subsystem  graphics  by  LRU.  Selecting  the  desired 
system/LRU  location  activates  the  link  and  takes 
the  user  to  the  appropriate  window. 

d.  Secondary  Menus.  Secondary  menus  should  be  in  a 
textual  format  offering  the  user  either  graphical 
or  direct  addressing  capability  and  should  lead 
the  user  to  either  descriptive  operational,  and 
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descriptive  general  information.  Additionally,  a 
textual  selection  should  be  offered  to  the  user  to 
perform  a  specific  function, 
e.  Menu  Layouts .  Menus  should  comply  with  previously 
defined  standards  for  textual  and  graphical 
representations.  Where  textual  menus  are  the  only 
content  in  a  window  they  should  be  either  centered 
vertically  on  the  display  screen  or  if  a  large 
number  of  options  are  available  (i.e.  greater  than 
25)  they  should  be  listed  vertically  down  the  left 
hand  side  of  the  window.  Where  menus  are 
displayed  with  explanatory  text  or  other 
amplifying  data  they  should  be  displayed 
horizontally  along  the  bottom  or  top  of  the 
screen.  Horizontal  menu  selections  should  be  non¬ 
scrolling  and  remain  visible  to  the  user  during 
multi-screen  scrolling  operations.  The  user 
selection  should  be  made  by  placing  the  cursor  on 
a  selection  and  pressing  either  the  enter  key  or 
left  pointing  device  (if  used)  button  to  activate 
the  link.  Each  menu  should  contain  an  option  to 
allow  the  user  to  exit  or  return  to  the  previous 
menu  level.  Graphical  representations  should  be 
centered  on  the  display  screen  and  provide 
sufficient  textual  explanation  at  the  bottom  of 
the  display  to  guide  the  user  in  its  use.  The 
graphical  representation  should  also  have  an 
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option  to  allow  the  user  to  return  to  the  previous 
menu  level  or  exit.  In  all  menus  textual  layouts 
and  explanations  should  comply  with  defined 
standards  and  formats  for  screen  layouts  and  be 
standard  explanatory  texts  to  facilitate  user- 
friendliness. 

f.  Tertiary  Menus.  Tertiary  menus  are  the  task 
performance  menus  and  should  offer  the  user  a 
graphical  and  textual  selection  interface  to 
enable  indication  of  the  LRU  to  perform  the  task 
on.  It  should  also  provide  the  user  with  all 
Descriptive  Operational,  Procedural  General  and 
Descriptive  General  data  for  the  selected  task. 

g.  Naming  Standards.  Menu  names  should  identify  the 
type  and  level  of  the  individual  menu.  At  the 
primary  menu  names  should  identify  the  menu  type 
and  vehicle  designation  e.g. : 

INITIAL  MENU  -  F15E 

The  secondary  menu  name  should  identify  the 
identify  the  selected  system  e.g.: 

NOSE  LANDING  GEAR  MENU 
The  tertiary  menu  name  should  identify  the 
function/task  and  the  system  name  or  S/S/SN  e.g.: 

INSTALLATION  -  NOSE  LANDING  GEAR. 

At  any  menu,  the  menu  name  should  not  only 
identify  the  function  of  that  particular  menu  but 
should  assist  the  user  in  determining  location 
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within  the  system.  An  additional  benefit  of 
standardized  menu  naming  formats  is  that  the  user 
is  more  easily  able  to  direct  address  access  the 
menu  by  name  rather  than  physically  tracking 
through  extra  nodes  to  locate  it. 

User  Interface  Sub-Model 

The  user  interface  is  one  of  the  most  important 
elements  of  any  model  design.  While  a  high  level  of 
technical  perfection  may  be  achieved,  the  inability  to 
transfer  information  to  the  intended  audience  in  a 
meaningful  and  understandable  manner  signifies  a  failure  to 
achieve  the  primary  purpose  of  the  application.  The  object 
of  any  information  storage  and  retrieval  system  is  to 
support  the  information  needs  of  its  users.  Thus,  it  should 
be  user  responsive  rather  than  system  responsive. 

While  there  is  no  one  presentation  method  that  is 
suitable  or  acceptable  for  all  users,  a  strategy  that 
maximizes  its  range  of  user  satisfaction  is  optimal.  There 
are  three  identified  levels  of  user  experience  that  need  to 
be  addressed:  the  inexperienced  new  user;  the  journeyman 
or  intermediate  experience  user;  the  expert  user. 

In  the  military,  experience  levels  can  be  equated  with 
rank,  thus,  the  inexperienced  new  user  would  equate  to  an 
enlisted  technician  having  recently  graduated  from  trade 
training  school.  The  journeyman  would  have  had  a  number  of 
years  experience  and  additional  specialist  training  and 
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achieved  up  to  the  rank  of  non-commissioned  officer  (NCO) . 
The  expert  equates  to  a  Senior  NCO  up  to  the  rank  of  Master 
Sergeant  (Warrant  Officer  in  the  Royal  Australian  Air  Force) 
with  significant  experience  on  a  range  of  vehicles. 

Knowledge  of  experience  levels  of  potential  users  helps 
to  ensure  that  an  appropriate  user  model  interface  is 
developed  to  properly  support  user  requirements.  The 
different  experience  levels  outlined  require  that  separate 
interfaces  be  modelled  for  each  with  the  greatest  assistance 
and  rigidity  given  to  the  inexperienced  user  the  greatest 
flexibility  to  the  expert.  While  each  experience  level 
should  have  its  own  access  method,  users  should  not  be 
locked  permanently  into  one  method  and  as  their  confidence 
and  ability  increases  selection  of  the  appropriate  access 
method  should  be  left  to  the  user. 

Another  element  impinging  on  design  strategy  is 
referencing  technique.  Individuals  identify  and  label 
objects,  thoughts  and  concepts  based  on  prior  knowledge, 
experience  or  socialization  experiences.  It  is,  therefore, 
possible  for  two  individuals  to  reference  the  same  object 
using  different  labels.  Referencing  is  important  in 
locating  information  as  indexing  strategies  used  are  based 
on  individually  allocated  labels.  The  problem  of  individual 
labelling  anomalies  is  addressed  in  printed  media  through 
the  use  of  "see”  and  "see  also"  cross-referencing  techniques 
and  the  use  of  synonymous  replicate  entries  in  indexes. 
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Any  indexing  system  developed  must  be  flexible  to  allow 
users  to  obtain  the  desired  information  quickly  and  with  a 
minimum  of  effort.  The  user  should  be  able  to  access  the 
task  or  any  of  the  LRU  specific  supporting  information  using 
either  a  menu  interface  or  a  direct  addressing  method. 

Inexperienced  users  should  be  provided  with  a 
hierarchical  menu  structure  that  will  lead  them  to  their 
desired  information,  reducing  the  potential  for  confusion 
and  frustration  in  not  being  able  to  use  the  system  as 
wished.  The  levels  of  experience  and  expertise  previously 
discussed  infer  increased  familiarity  with  terminology  and 
labels  commensurate  with  levels  of  experience.  It  therefore 
follows  that  the  journeyman  and  expert  user  should  be 
provided  with  an  indexing  and  retrieval  interface  that 
recognizes  this  increased  familiarity  and  allows  direct 
specification  of  desired  information.  In  either  case  the 
menu  structure  and  direct  addressing  should  be  options 
available  to  all  users  with  the  individual  choosing  the 
desired  method  based  on  a  self-assessment  of  needs  and 
abilities . 

The  success  of  direct  addressing  is  dependent  upon  a 
standardized  naming  convention  for  the  individual  data 
modules.  A  consistent  strategy  should  allow  the  experienced 
user  and  journeyman  to  recognize  information  elements 
through  the  of:  common  names  and  S/S/SN's  in  the  case  of 
LRU's;  names  that  reflect  the  nature  of  the  task  being 
performed;  in  the  case  of  explanatory  information,  the  type 
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of  explanation  and  S/S/SN;  names  for  graphics  that  indicate 
the  S/S/SN  and  type  of  graphic. 

Data  Sub-Model 

Analysis  of  the  current  F15E  technical  maintenance 
publication  set,  Appendix  A,  has  led  to  the  categorization 
of  five  distinct  data  groups.  These  are; 

a.  Authoritative  data 

b.  Descriptive  General  data 

c.  Procedural  General  data 

d.  Descriptive  Operational  data 

e.  Procedural  Operational  data 

The  SHADM  model  is  structured  around  the  performance  of 
seven  primitive  tasks  and  the  information  contained  in  each 
data  group  has  been  segmented  according  to  its  relationship 
to  each  task  and  the  vehicle  and  how  it  supports  the 
performance  of  the  task.  The  data  sub-model  so  derived  is 
detailed  in  Figure  4. 

Authoritative  Data.  The  Authoritative  data  group  is 
comprised  of  two  elements,  implied  authority  and  specified 
authority.  Implied  authority  is  vested  in  the  publication 
set  in  that  it  implicitly  grants  authority  for  performance 
of  the  tasks  and  procedures  contained  in  the  manuals. 
Specified  authority  is  contained  in  the  Time  Compliance 
Technical  Orders  which  authorize  specified  actions  with 
specified  time  frames. 
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Authoratative 


Figure  4:  SHADM  Data  Sub-Model 
specified  time  frames. 

Descriptive  General  Data.  Descriptive  General  data  is 
made  up  of  data  elements  that  apply  to  all  support  aspects 
of  technical  maintenance  and  are  not  task  performance 
specific.  These  include  the  following  elements: 

a.  Symbols  and  abbreviations 

b.  General  aircraft  descriptions 

c.  Safety  and  protective  devices 

d.  Emergency  procedures 

e.  Placards  and  markings 

f.  Dimensions  and  Areas 
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application  across  the  whole  vehicle  data  set.  They  are 
displayed  in  either  definitional  text,  verbose  text  rt 
graphical  format. 

Procedural  General  Data.  Procedural  General  data  is 
made  up  of  procedure  or  task  specific  information  that 
relates  to  the  performance  of  more  than  one  task  on 
systems/subsystems/LRU's.  It  is  more  specific  than 
Descriptive  General  as  it  is  procedural  in  its  orientation 
and  its  key  identifier  is  its  data  content  support  for 
multiple  systems.  Procedural  General  data  includes  the 
following  elements: 

a.  General  maintenance 

b.  Consumable  materials 

c.  Support  equipment 

d.  Jacking  and  lifting 

e.  Levelling 

f.  Aircraft  towing 

g.  Parking  and  Mooring 

h.  Servicing 

As  these  data  components  are  of  a  general  nature  they  are 
globally  defined  within  the  model  for  application  across  the 
entire  vehicle  procedural  data  set.  They  are  displayed  in 
either  definitional  text,  verbose  text  or  graphical  format. 

Descriptive  Operational  Data.  Descriptive  Operational 
data  relates  to  the  performance  of  specific  tasks  on 
individual  systems,  subsystems  or  LRU's.  This  data  type  is 
unique  to  each  unit  and  as  such  is  not  able  to  be  globally 
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defined,  rather,  it  must  be  locally  defined.  The  data  type 
includes : 

a.  System  functional  description 

b.  Special  maintenance  requirements 

c.  Principles  of  operation 

Descriptive  Operational  data  is  presented  in  the  textual 
verbose  mode  only. 

Procedural  Operational  Data.  Procedural  Operational 
data  is  the  task  specific  information  used  in  the 
performance  of  the  seven  primitive  tasks  on  systems, 
subsystems  and  LRU's.  It  is  that  information  that  is  unique 
to  both  task  and  system  and  as  such  is  not  suitable  for 
global  definition.  It  is  displayed  in  either  definitional 
text,  verbose  text  or  graphical  format. 

Data  Relationship  Sub-Model 

The  data  relationship  sub-model  examines  the  various 
data  elements  in  the  current  technical  maintenance 
publication  system  and  defines  the  relationships  between 
these  elements.  These  relationships  become  the  hypermedia 
links.  The  SHADIn  model  utilizes  the  following  linking 
strategies : 

a.  links  to  text 

b.  links  to  graphics 

The  textual  links  result  in  the  foLlowing  display  formats: 
a.  text  expansion  -  where  the  existing  text  i: 

expanded  through  the  insertion  of  additional  text. 
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This  additional  text  is  removed  from  the  display 
when  the  link  is  selected  again. 

b.  new  window  text  -  where  the  text  window  displayed 
is  replaced  by  a  new  text  window. 

c.  text  sub-window  -  where  a  smaller  window  opens 
within  the  current  display  window  to  provide 
amplifying  or  explanatory  text. 

The  data  relationship  sub-model  will  be  developed  in 
two  stages:  firstly,  the  elements  that  are  Descriptive 
General  and  Descriptive  Operational  and  support  the  vehicle 
and  its  systems  rather  than  the  individual  tasks  will  be 
examined;  secondly,  the  Procedural  Operational  and 
Procedural  General  segments  will  be  developed. 

Analysis  of  the  current  publication  set,  Appendix  A, 
has  revealed  that  all  data  available  in  the  manuals  supports 
the  performance  of  seven  (7)  primitive  tasks.  These  are: 

a.  Removal 

b.  Installation 

c.  Inspection 

d.  Cleaning 

e.  Rigging  and  Operational  Checkout 

f.  Adjustment 

g.  Calibration  (4:1-1) 

In  their  current  format  these  tasks  form  a  complex  set 
of  relationships  which  exist  within  each  level  and  between 
each  level  of  the  publication  set  as  well  as  in  the 
performance  of  the  different  functions  on  the  various 
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systems,  and  subsystems  strata.  This  complication  and 
subsequent  necessity  for  the  user  to  be  completely  familiar 
with  the  publication  set  is  a  function  of  the  media  and  not 
the  data  or  the  task.  As  previously  identified,  the  data 
supports  the  performance  of  seven  primitive  tasks  on  the 
vehicle  and  its  components  thus,  it  is  suited  to  a  task 
oriented  structure.  This  is  not  possible  with  the  present 
manual  system  as  the  volume  of  relationship  cross-references 
would  be  difficult,  if  not  impossible  to  elucidate,  and 
would  result  in  an  incomprehensible  document.  This  data 
interaction  complexity  has  been  addressed  to  some  extent 
through  the  use  of  the  three  tiered  publication  system 
currently  in  place. 

The  Job  Guide  details  functions  to  be  performed  on 
system,  subsystem  and  sub-subsystem  LRU's  (21:8).  McDonnell 
Douglas  define  three  specific  elements  to  the  operations 
described  in  the  JG  set: 

a.  step  -  "a  single  and  separate  maintenance  action" 
(21:8) . 

b.  task  -  "a  series  of  steps  leading  to  the 
accomplishment  of  one  of  the  procedures"  (21:8). 

c.  function  -  "is  composed  of  one  or  more  tasks" 

(21:8). 

Conceptually,  these  definitions  are  represented  in  Figure  5. 

Each  of  the  tasks  defined  for  a  function  represent  one 
of  the  seven  primitive  tasks  identified  earlier.  As  can  be 
seen  from  the  diagram,  each  step,  task  and  function  (not 
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Figure  5:  Operational  Elements 

Each  of  the  tasks  defined  for  a  function  represent  one 
of  the  seven  primitive  tasks  identified  earlier.  As  can  be 
seen  from  the  diagram,  each  step,  task  and  function  (not 
shown)  are  not  mutually  exclusive  but  share  common 
information  relationships.  Although  the  relationships 
identified  through  analysis  are  significantly  more  complex 
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procedural  elements.  Dissecting  the  structure  to  the 
procedural  element  level  would  create  a  model  or 
representation  of  such  complexity  that  it  would  be  unusable. 

In  its  present  structure  the  task  orientation  of  the 
publication  set  is  based  on  the  performance  of  the  primitive 
maintenance  procedures  on  each  LRU.  Thus,  the  publication 
set  at  the  JG  level  potentially  has  seven  times  the  total 
number  of  vehicle  LRU  instantiations.  Transferring  this 
structure  to  electronic  format  as  it  is  would  not  be 
feasible  due  to  the  large  demand  it  would  place  on  both 
primary  and  secondary  data  storage  resources.  Additionally, 
updating  and  deleting  data  would  be  difficult  and  time 
consuming  due  to  the  potentially  large  number  of 
replications . 

Identifying  the  primitive  tasks,  installation,  removal, 
etc,  as  the  tasks  for  orientation,  rather  than  a  specific 
LRU  installation  or  LRU  removal,  allows  the  segmenting  of 
common  global  procedures  from  the  LRU  specific  ones. 
Graphically  this  is  shown  in  Figure  6.  Thus  the 
relationships  are  on  the  basis  of  common  to  uncommon,  where 
common  data  relates  to  more  than  one  LRU  while  uncommon  is 
single  LRU  specific  in  nature.  This  will  result  in  a 
significantly  smaller  physical  data  base.  Data  maintenance 
will  be  enhanced  as  changes  would  affect  either  common  data 
(task)  or  specific  data  (LRU)  each  contained  in  separate 
files  or  nodes. 
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Figure  6:  Primitive  Task  Orientation 

will  be  enhanced  as  changes  would  affect  either  common  data 
(task)  or  specific  data  (LRU)  each  contained  in  separate 
files  or  nodes. 

Removing  common  data  from  the  LRU  nodes  will  allow  for 
global  definition  strategies  and  multiple  node  links  to  be 
used  thus  further  reducing  overall  document  size.  Global 
definition  is  where  data  elements  and  their  definitions  or 
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link  from  multiple  files  that  contain  references  to  data  or 
require  amplification  by  the  target  data  stored  in  that 
node.  This  element  of  the  model  is  considered  essential  for 
robustness  as  it  will  significantly  limit  the  potential  for 
deletion  and  update  anomalies  during  data  modification  as 
commonly  referenced  data  are  located  in  single  files. 

Descriptive  General  data  supports  the  task  performance 
by  providing  technical  information  that  is  not  specific  to 
an  individual  function  or  task  on  a  system,  subsystem  or  LRU 
but  refers  generally  to  the  overall  vehicle,  system, 
subsystem,  or  LRU. 

The  general  descriptive  nature  of  this  data  type 
produces  relationships  with  other  elements  of  the  data  set 
in  a  supportive  role.  This  data  has  three  formats  and  each 
should  have  a  linking  strategy  that  recognizes  both  its  role 
and  format.  The  formats  are: 

a.  Descriptive  verbose 

b.  Definitional 

c.  Graphical 

Descriptive  Verbose.  Descriptive  verbose  data  provides 
information  to  the  user  in  a  textual  format  and  can  range  in 
size  from  one  line  to  any  number  of  paragraphs.  This  type 
of  data  requires  a  separate  window  and  the  linking  strategy 
is  to  use  a  link  to  text  in  a  new  window. 

Definitional .  This  data  type  usually  comprises  one  or 
two  lines  and  provides  explanatory  information  about  a  word, 
phrase  or  graphic  element.  Due  to  its  amplifying  nature  its 
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link  type  should  be  a  text  link  within  the  same  window  but 
using  either: 

a.  a  technique  that  physically  expands  the  selected 
text  by  placing  the  definition  data  adjacent  to 
the  current  data. 

b.  displaying  the  data  in  another  location  in  the 
display  window. 

Returning  from  the  link  should  remove  the  definitional  text 
and  leave  the  display  screen  as  it  was  prior  to  the  link 
selection. 

Graphical .  Graphical  data  requires  a  separate  display 
window  to  ensure  there  is  no  data  loss  due  to  physical  size 
reduction.  The  link  should  be  a  graphics  link  to  a  new 
window,  thus  totally  replacing  the  data  in  the  window  where 
the  link  was  selected  with  the  chosen  graphic. 

Procedural  General  and  Procedural  Operational  data  are 
those  data  elements  that  specifically  support  the 
performance  of  the  seven  primitive  tasks,  previously 
defined,  on  the  system,  subsystem  or  LRU. 

Procedural  General  data  perform  the  same  supportive 
function  as  the  Descriptive  General  data  although  possessing 
a  more  specific  nature.  Rather  than  providing  global  data 
definition,  the  procedural  data  directly  relates  to  the 
performance  of  tasks  on  the  vehicle  elements.  It  has  the 
same  formats  and  relationships  as  the  Descriptive  General 
data  and  as  such  uses  the  same  linking  strategies. 
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Procedural  Operational  data  are  data  elements  that  are 
task  performance  specific.  They  have  the  same  formats  and 
relationships  as  the  Descriptive  General  data  although  their 
display  requirements  differ.  The  graphical  elements  provide 
visual  prompts  to  the  user  in  task  completion  while  the 
descriptive  verbose  provide  a  textual  description  of  the 
task  steps. 

The  linking  strategies  used  with  Procedural  Operational 
data  differ  from  the  other  data  elements  in  that  the  user 
requires  the  graphical  data  to  be  constantly  displayed  for 
referential  purposes  concurrently  with  textual  explanation 
of  steps  and  definitions.  To  avoid  conflict  between  both 
formats  the  model  displays  them  simultaneously,  when 
required,  but  in  a  manner  that  eliminates  interference 
between  the  two.  The  SHADM  model  displays  the  graphical 
data  as  the  main  display  with  callouts  to  the  procedural 
steps,  numbered  in  task  completion  sequence.  Links  between 
the  procedural  step  numbers  are  text  links  to  a  boxed 
display  at  the  top  of  the  display  page.  Text  scrolling  is 
required  within  the  link  display  where  the  text  size  is 
larger  than  the  display  box  to  avoid  the  central  graphic 
being  overwritten  by  text.  This  linking  and  display 
strategy  ensures  that  the  user  is  able  to  visually  access 
both  information  formats  simultaneously. 

Definition  displays  resulting  from  global  linkages  to 
descriptive  support  data  utilize  the  linkage  mode  detailed 
for  definitional  descriptive  general  data. 
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Summary 


In  Chapter  IV  the  SHADM  model  was  developed  based  on 
the  consensus  direction  provided  by  the  findings  of  the 
literature  review  conducted  in  Chapter  II  and  the  detailed 
analysis  of  the  F-15E  Technical  Maintenance  manual  set  in 
Chapter  IIT.  Four  major  areas  of  significance  were 
addressed  in  the  design  phase: 

a.  Standards  and  formats  -  these  detailed  how  data 
should  be  presented  to  the  user 

b.  User  interface  sub-model  -  which  provided 
alternate  methods  of  using  the  system  based  on 
user  experience  levels 

c.  Data  sub-model  -  defined  the  logical  structuring 
and  segmenting  of  the  overall  data  set 

d.  Data  relationship  sub-model  -  which  provided  the 
style  of  the  links  to  be  created  between  the 
logical  data  segments  based  on  content 
relationships  to  the  task  orientation  of  the  model 
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V.  Findings  and  Analysis 


The  objective  of  this  chapter  is  to  detail  the  SHADM 
model  prototype  development  and  testing  procedures 
undertaken,  and  examine  the  test  results.  To  achieve  this, 
the  following  areas  will  be  examined: 

a.  Prototype  development 

b.  Software  selection 

c.  Model  testing 

d.  Test  results 

Prototype  Development 

Preparation  of  the  prototype  system  was  completed  using 
the  following  commercial  software  packages; 

a.  HyperWriter  -  hypermedia  application 

b.  Omnipage  Professional  -  optical  character 
recognition  a’^plication 

c.  Microsoft  Pail  brush  -  graphics  display  and 
manipulation  application 

d.  Chinon  Deskscan  2000  -  scanning  hardware  and 
software 

The  prototype  design  rigidly  followed  the  standards  and 
constructs  detailed  in  Chapter  IV,  with  the  exception  of  the 
graphics  display  and  storage  formats.  Graphics  were  scanned 
as  raster  images  in  accordance  with  MIL-STD-1840A,  however, 
they  were  converted  to  .PCX  format  for  storage  and  display 
purposes.  Additionally,  a  HyperWriter  generated  graphics 
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forms,  .MGR,  was  also  used  in  one  file  application.  This 
divergence  from  model  requirements  for  conformance  with  MIL- 
STD  specifications  resulted  from  software  rather  than  model 
limitations  and  is  not  considered  to  have  effected  the 
validity  of  final  testing  results. 

The  second  divergence  from  standards  was  in  the 
presentation  style  of  graphics  and  text  at  the  task  level. 
Two  methods  were  offered  to  the  users  for  evaluation,  the 
first  being  graphic  to  text  format.  This  method  displayed 
the  illustration  and  had  embedded  links  on  the  relevant 
graphic  elements  relative  to  the  required  procedural  steps. 
Selecting  these  links  resulted  in  the  required  step  text 
being  displayed  in  a  window  in  the  top  right  hand  corner  of 
the  screen.  The  effect  of  this  approach  was  that  both  text 
and  graphic  information  were  visible  simultaneously  thus 
allowing  the  user  to  read  the  text  while  retaining  visual 
reference  to  the  part  being  worked  on. 

The  second  option  offered  was  a  text  to  graphic 
approach  whereby  the  user  entered  a  full  text  screen 
detailing  individual  procedural  steps  in  full.  Each  step 
was  linked  to  its  relevant  illustration.  Link  activation 
resulted  in  the  text  screen  being  replaced  with  the  full 
page  graphic  screen  containing  the  illustration.  Additional 
textual  support  was  not  provided  nor  were  there  further 
external  links.  Reference  to  the  procedural  text  required 
deactivation  of  the  link. 
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The  prototype  took  approximately  60  hours  to  complete 
and  was  structured  around  the  Nose  Landing  Gear  (NLG)  system 
of  the  F-15E  aircraft.  Approximately  50%  of  the  General 
Vehicle  Manual  was  converted  to  hypertext  and  linked  using 
the  model  relationship  criteria.  All  NLG  specific 
information  from  the  General  Service  Manual  was  converted, 
with  the  exception  of  graphics.  The  maintenance  procedure 
selected  for  display  was; 

NLG  UPPER  JURY  LINK  REMOVAL,  INSTALLATION,  AND 

OPERATIONAL  CHECKOUT 

The  prototype  included  all  procedural  support  data, 
maintenance  steps  and  graphical  representations.  The  final 
package  was  comprised  of  one  ASCII  text  file,  22  .PCX  format 
graphics  files  and  one  HyperWriter  generated  .MGR  format 
graphics  file. 

Software  Selection 

HyperWriter  was  selected  as  the  software  for  use  in  the 
prototype  modeling,  however,  this  should  not  be  seen  as  a 
recommendation  for  the  use  of  this  package  over  any  other. 
Time  limitations  imposed  by  academic  requirements  denied  the 
ability  to  rigorously  and  extensively  evaluate  all 
commercially  available  hypermedia  packages. 

The  current  version  of  HyperWriter  does  not  meet  the 
requirements  of  MIL-STD's  in  that  it  is  unable  to  handle  the 
storage,  retrieval  and  presentation  of  either  Computer 
Graphic  Metafile  (CGM)  or  Initial  Graphics  Exchange 
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Specification  (IGES)  graphic  formats.  Given  this  limitation 
it  did  meet  the  selection  requirement  parameters  of: 

a.  Ease  of  use 

b.  User  friendly 

c.  Fast  response  times 

d.  Easy  development  environment 

e.  Uncomplicated  document  navigation  capability 

f.  Ability  to  concurrently  display  graphics  and  text 

g.  Boolean  search  capability 

Two  other  commercial  hypertext  packages  were  subjected 
to  limited  evaluation,  limited  by  the  capabilities  of  the 
demonstration  packages  provided  by  the  manufacturers,  rather 
than  the  depth  of  evaluation.  The  products  were: 

a.  Knowledgepro  from  Knowledge  Garden  Inc 

b.  Guide  by  Owl  International 

Knowledgepro  was  not  considered  suitable  due  to  the 
slow  speed  with  which  it  activated  links  and  its  screen 
refresh,  despite  being  tested  on  an  80386/25Mhz  computer. 
Additionally,  it  possessed  a  complex  programming  language 
and  was  orien^'ed  towards  Expert  Systems  development  rather 
than  Hypermedia. 

Owl  was  also  found  to  be  unsuited  to  the  technical 
maintenance  documentation  model.  Despite  having  a  well 
developed  graphics  display  and  handling  capability,  it  did 
not  provide  the  required  hypermedia  support,  was  difficult 
to  program  in  and  its  navigational  capabilities  were 
1 imited . 
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Model  Testing 


Model  prototype  testing  was  conducted  at  Wright 
Patterson  AFB  utilizing  personnel  from  the  F-15E  Systems 
Project  Office,  C-17  Systems  Project  Office,  International 
Logistics  Office  and  the  Air  Force  Institute  of  Technology 
(AFIT) .  The  objectives  of  the  testing  were  to  establish  the 
validity  of  the  proposed  technical  maintenance  publication 
model  through  specific  field  examination  of  the  following 
areas : 

a.  User  interface  -  how  user  friendly  and  easy  the 
system  is  to  understand  and  use. 

b.  Display  standards  -  are  the  screen  formats/layouts 
appropriate  for  the  information  content  being 
presented  and  is  the  model  logically,  structured. 

c.  Usability  -  how  robust  is  the  model  when  placed  in 
a  working  environment  and  subjected  to  non¬ 
developer  use. 

Testing  was  conducted  unsupervised  with  each 
participant  being  provided  with  minimal  instruction  on  the 
syscem  use.  Instructions  were  verbal  and  were  limited  to: 
starting  the  package;  the  concept  of  lj.nks;  how  to  activate 
and  deactivate  links;  explanation  of  the  two  graphic  display 
formats;  how  to  exit  the  package.  Instruction  was 
deliberately  restricted  as  a  further  validation  test  of  the 
user  friendliness  and  ease  of  model  use. 
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User  feedback  was  obtained  using  a  questionnaire  which 
addressed  the  specific  areas  of  interest.  A  copy  of  the 
questionnaire  is  at  Appendix  B. 

Test  Results 

Testing  of  the  SHADM  model  prototype  wns  con  cted  over 
a  two  week  period  and  of  nine  participants  selected  for 

the  evaluation,  eight  returned  completed  questionnaires  at 
the  end  of  the  testing  period.  These  responses  were 
consolidated  and  the  results  are  detailed  at  Appendix  C. 

Analysis  of  th  results  was  conducted  in  two  parts,  a 
general  overview  analysis  of  each  general  area  tested  and  a 
detailed  question  by  question  analysis  is  provided  at 
Appendix  D. 

Scoring  Evaluations.  Scoring  of  the  evaluation  test 
was  done  using  a  10  point  scale  with  users  indicating  their 
]  :Vcl  of  agreement  with  the  listed  statements.  A  rating  of 
Id  indicated  strong  agreement  while  a  rating  of  0  indicated 
a  strong  disagreement.  These  scores  were  evaluated  on  the 
basis  of: 

a.  0  -  totally  unacceptable 

b.  1-4  -  unacceptable 

c.  5  -  acceptable 

d.  6-7  -  good 

e.  8-9  -  very  good 

f.  10  -  excellent 
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User-Interface .  Questions  1  to  5  addressed  aspects  of 
the  model  relative  to  the  user-interface.  The  objective  was 
to  determine  how  friendly  the  system  was  to  the  user  and  how 
much  it  assisted  the  user  to  operate  the  system  effectively. 
Overall,  this  segment  received  an  average  score  of  8.37  from 
a  range  of  7.625  to  9.  This  indicates  a  very  good  user- 
interface.  Individual  comments  addressed  the  need  for 
supporting  user  documentation  and  there  were  a  number  of 
complimentary  comments  that  provided  no  additional  model 
developmental  support. 

Display  Standards.  Model  display  standards  were 
addressed  by  questions  6  to  16  and  covered  the  areas  of  text 
and  menu  displays,  and  graphic  presentations.  Overall  users 
rated  the  display  standards  as  good  with  an  average  score  of 
7.74.  This  lower  score  was  qualified  with  pertinent 
comments  regarding  the  colors  to  use  on  graphical  displays, 
black  on  white;  the  need  to  display  menu  buttons  in  a 
standard  length  for  uniformity;  a  preference  for  both  text 
to  graphic  and  graphic  to  text  displays  with  both  elements 
displayed  simultaneously;  the  need  for  the  user  to  reference 
multiple  screens  for  schematic  and  wiring  data  fault 
finding;  improvement  in  graphics  display  quality;  provision 
of  proper  user  documentation  to  support  on-line  access.  The 
comments  offered  indicated  that  additional  support  of  the 
system  was  required  in  the  form  of  training  and  user 
documentation.  This  particular  aspect  is  not  considered  an 
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element  of  the  SHADM  model  per  se,  rather  an  implementation 
strategy. 

General .  Questions  17  to  20  addressed  general  issues 
relative  to  the  acceptability  and  usability  of  the  model  in 
a  working  environment.  The  average  score  of  7.34  rates  this 
as  good  and  the  additional  comments  raised  issues  of 
concern,  although  not  all  related  to  the  model  design.  In 
particular,  failure  of  the  tested  system  to  meet  military 
specifications  resulted  in  some  significantly  lower  scores, 
however  this  issue  has  already  been  addressed  in  the  model 
design  standards  and  further  discussion  is  not  warranted. 
Hardware  suitability  and  availability  was  another  issue  that 
concerned  the  users  and  although  this  does  not  warrant 
consideration  in  this  study,  it  is  certainly  an  issue  that 
demands  attention  prior  to  any  implementation  decisions. 

Model  Modifications 

The  model  testing  resulted  in  a  number  of 
recommendations  from  the  users  for  improvement  of  the  model. 
These  were: 

a.  Standardization  of  menu  button  sizes 

b.  Provision  of  both  text  to  graphic  and  graphic  to 
text  for  Procedural  Operational  data  and 
illustration  displays 

c.  Use  of  markers  for  quick  multiple  window  cross- 
referencing 


62 


Each  of  these  recommendations  will  be  incorporated  in  the 
SHADM  model. 

Summary 

This  chapter  addressed  the  development  of  the  SHADM 
prototype  model,  identified  the  software  and  hardware  used 
to  create  the  prototype,  examined  the  software  selection 
criteria  and  justified  final  presentation  software 
selection.  Model  testing  was  conducted  using  personnel  with 
a  diverse  range  of  backgrounds  including  military  aircraft 
technicians,  military  engineers  both  domestic  and  foreign, 
and  civilian  contractor  publication  specialists.  Overall 
the  test  results  rated  the  SHADM  model  as  being  good  and 
three  modification  recommendations  received  from  the 
evaluation  panel  will  be  incorporated  in  the  model. 
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VI .  Conclusions  and  Recommendations 


This  chapter  details  the  writers’  conclusions  regarding 
the  objective  of  this  research,  as  defined  in  Chapter  I,  and 
recommendations  for  future  development  of  the  SHADM  model. 

Publications 

Determination  of  how  an  RAAF  technical  maintenance 
publication  could  be  converted  to  a  Hypermedia  application 
and  to  design  and  test  a  Hypermedia  model  was  the  stated 
objective  of  this  research.  In  approaching  this  task 
definitions  of  RAAF  technical  maintenance  publications  and 
Hypermedia  applications  were  presented. 

Detailed  analysis  of  the  existing  F-15E  maintenance 
publication  set  revealed  a  tiered  structure  based  around  the 
performance  of  functions  on  specific  LRU's.  This  was 
identified  as  a  function  of  the  delivery  media,  books, 
rather  than  the  logical  structure  of  the  data  content. 
Content  was  determined  to  be  task-oriented,  however, 
modification  was  required  in  the  underlying  structure  from  a 
task  LRU  specific  to  task  specific  orientation. 

A  data  structure  was  extracted  from  the  existing  data 
set  revolving  around  three  data  types  which  supported  the 
required  task-orientation  for  Hypermedia  compatibility. 

These  data  types  were  further  stratified  into  five  distinct 
groups  which  supported  each  task.  These  groups  were: 

a.  Authoritative 
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b.  Descriptive  General 

c.  Procedural  General 

d.  Descriptive  Operational 

e.  Procedural  Operational 

Natural  relationships  between  the  data  groups  and  the 
individual  tasks  provided  the  basis  of  a  linking  strategy. 

In  this  format  the  existing  publication  set  was  readily 
transitioned  into  a  robust  Hypermedia  application. 

Model 

Model  creation  on  the  basis  of  the  publication  data  set 
analysis  centered  around  meeting  the  criteria  identified  in 
the  literature  review  relative  to  user-interaction.  This 
was  a<.’hieved  through  extensive  use  of  graphical 
representa^  Lons  and  a  tiered  menuing  strategy.  Three  levels 
of  us'^tr  were  idem  fied,  ew,  intermediate,  and  expert,  and 
a  strategy  developed  for  each  based  on  their  experience  and 
anticipated  vehicle  familiarity.  There  is  a  temptation  for 
developers  to  over-emphasize  the  graphical  presentations, 
resulting  in  cluttered  displays.  In  this  respect  the 
Hypei Writer  software  imposed  no  limitations  and  a 
disciplined  approach  to  presentation  design  was  essential. 

Segmentation  of  existing  data  was  rapidly  achieved  by 
applying  the  model  criteria  to  each  section.  The  most 
difficult  and  time  consuming  element  was  converting  the  text 
into  electronic  format.  The  quality  of  existing 
publications,  relative  to  font  and  typeface  size,  made  it 
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difficult  to  effectively  utilize  the  OCR  software  acquired 
for  the  purpose,  however,  it  significantly  reduced  the 
required  effort.  Given  the  capabilities  of  electronic  data 
transfer  required  under  CALS,  this  is  not  seen  as  a  major 
problem  in  future  conversion  or  in  the  development  of  new 
applications.  The  ease  of  data  conversion  using  the  model 
validates  its  inherent  logic  and  structure,  and  supports  the 
conclusion  that  it  is  an  appropriate  model  on  which  to  base 
Hypermedia  design  of  technical  maintenance  publications. 

This  conclusion  was  further  validated  by  the  results 
and  recommendations  of  the  prototype  evaluators  who  rated 
the  model  in  three  areas;  user-friendliness  which  addressed 
the  presentation  formats  and  layouts;  data  structures  which 
examined  the  data  segmentation,  logical  flow  and  linking 
strategies  implemented;  general  applicability  in  working 
environment  use  and  publications  management.  Cosmetic 
change  recommendations  were  made  but  overall  the  users 
declared  the  model  as  being  good  and  easy  to  use. 

Cost  Benefits 

Without  doubt  one  of  the  most  significant 
justifications  for  implementing  new  systems  is  a  reduction 
in  expenditure  over  the  existing  system.  This  is  even  more 
so  given  the  present  economic  climate  relative  to  military 
spending.  Use  of  the  SHADM  model  and  electronic  management 
of  technical  maintenance  publications  data  will  result  in 
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both  financial  and  manpower  savings  in  overall  production, 
distribution,  maintenance,  and  storage. 

Production  savings  will  be  realized  through  replacement 
of  the  current  paper  based  hard  copy  publication  set  with 
electronically  prepared  and  stored  technical  data.  The 
present  system  requires  the  preparation,  proof  reading, 
printing  and  collation  of  the  paper  manuals.  Under  an 
electronic  Hypermedia  system  these  processes  would  become 
redundant  as  the  technical  data  would  be  prepared  and 
transmitted  to  the  user  electronically. 

Publication  maintenance  becomes  significantly  easier  in 
an  electronic  environment  as  the  need  to  provide  manpower 
and  time  to  physically  incorporate  publication  amendments 
and  updates  is  negated.  Technicians  could  be  released  for 
primary  task  performance  rather  than  publication 
maintenance.  Additionally,  the  costs  associated  with  bulk 
handling  and  postage  of  publication  and  amendment 
distributions  would  be  reduced  to  an  insignificant  amount, 
sufficient  to  cover  the  expenses  associated  with  either 
floppy  disk,  tape  cartridge  or  compact  disc  distribution, 
depending  upon  the  media  transfer  strategy  adopted. 

Further  cost  savings  would  be  realized  through  removal 
of  the  need  to  bulk  store  excess  publication  and  amendment 
copies  for  future  use.  The  master  copies  would  be  stored 
electronically  thus  releasing  expensive  warehouse  capacity 
and  personnel  for  reallocation. 
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General 


This  research  project  has  resulted  in  the  development 
of  a  robust,  user-friendly  technical  maintenance  publication 
model.  This  has  been  achieved  in  the  model  design  through 
the  use  of  standardized  formats  and  presentation  style, 
multiple  data  accessing  techniques  that  recognize  different 
levels  of  user  experience,  the  use  of  uncluttered  screen 
presentations,  and  the  structuring  of  the  data  presentation 
to  a  task  orientation  format. 

The  analysis  conducted  on  existing  publications  and 
Hypermedia  applications  shows  that  these  manuals  are  suited 
to  conversion  to  Hypermedia.  The  information  content  has 
distinct  and  identifiable  logical  links  based  on  data  types 
which  support  the  completion  of  seven  distinct  primitive 
tasks.  Data  replications,  both  textual  and  graphical,  that 
exist  as  a  result  of  the  existing  presentation  format  can  be 
easily  removed  to  streamline  data  storage  and  presentation. 
The  model  developed  was  prototyped  and  tested  by  experienced 
specialist  users  who  validated  it  indicating  that  technical 
maintenance  publications  can  be  successfully  converted  to  a 
Hypermedia  application  using  the  SHADM  model. 

Existing  RAAF  technical  maintenance  publications,  are 
similar  to  USAF  publications  in  that  weapons  systems  and 
delivery  vehicles  used  by  both  services  come  from  the  same 
sources,  and  similar  aircraft  are  employed  by  both  air 
forces.  While  some  publication  conversions  are  made  to  a 
unique  RAAF  format,  in  most  instances  the  manuals  are 
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utilized  in  their  original  format  with  only  RAAF  unique 
amendments  being  incorporated  through  additional  page 
insertions.  This  similarity  and  the  validation  of  the  SHADM 
prototype  clearly  indicates  that  these  publications  can  be 
successfully  converted  using  the  SHADM  model.  However, 
conversion  of  all  publication  sets  will  not  result  in 
savings  and  careful  analysis  of  cost  benefit  against  vehicle 
life  cycle  and  conversion  costs  will  be  essential  prior  to 
implementation . 

Recommendations 

The  RAAF  has  been  managing  its  technical  maintenance 
publications  in  paper  media  since  its  inception  in  1921. 
While  production  methods  have  changed  and  developed  over  the 
years,  the  basic  information  delivery  method  has  not. 
Technology  in  hardware  and  software,  and  user  knowledge  and 
experience  has  now  reached  a  stage  where  it  can  adequately 
support  the  new  devel  mental  step  in  information 
management,  electro  anagement.  The  SHADM  model 

developed  through  t  research  is  considered  a  suitable 
basis  on  which  to  begin  the  transition. 

The  SHADM  model  has  been  validated  through  prototype 
testing  and  the  objective  of  this  research,  how  an  RAAF 
technical  maintenance  publication  can  be  transformed  into  a 
task-oriented  Hypermedia  application,  and  to  develop  a  model 
that  can  be  used  for  future  transformations,  has  been 
achieved.  The  prototype  developed  was  a  limited  application 
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of  the  model,  limited  by  restrictions  addressed  elsewhere  in 
this  paper,  and  while  this  does  not  detract  from  the 
efficacy  of  the  underlying  design  and  structure,  it  does 
demand  further  extensive  and  rigorous  testing  prior  to 
adoption.  The  critical  nature  of  the  application,  aircraft 
maintenance,  demands  that  a  full  working  application  be 
developed  and  tested  under  operational  conditions  before  any 
implementation  decision  be  made.  This  testing  should 
address  the  issues  of  model  reliability,  accuracy  and 
robustness  and  provide  a  test  bed  for  evaluation  of  delivery 
strategies,  hardware  systems,  and  user  acceptance. 

Extensive  evaluation  of  available  Hypermedia  packages 
should  be  conducted  to  ensure  selection  of  the  optimal 
application  to  meet  model,  existing  CALS  specifications,  and 
end  user  requirements. 

Future  Directions 

Future  research  directions  should  be  focused  on 
maximizing  the  utility  of  the  elements  contained  in 
Hypermedia  packages.  The  integration  of  these  components 
should  result  in  significant  synergistic  benefits. 

Lateral  development,  utilizing  the  flexibility  of 
Hypermedia  applications,  into  expert  system  for  fault 
isolation  and  diagnosis  should  be  the  subject  of  further 
research.  Tnis  extension  should  also  be  applied  to  the 
integration  of  supply  systems  allowing  technicians  to 
identify  defective  parts  in  the  procedural  phase  of  the  task 
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completion  and  interrogate  the  supply  database  for  spares 
availability  and  to  lodge  order  requests  without  the  need 
for  paperwork. 

Training  is  another  aspect  of  this  development  that 
offers  significant  potential  benefits.  Yet  another  element 
of  Hypermedia,  the  ability  to  incorporate  video  segments, 
would  allow  the  development  of  interactive  training  packages 
that  would  provide  the  ability  for  trainees  to  engage  in 
self-paced  education.  This  particular  application  could 
also  be  incorporated  in  the  maintenance  publication  model  to 
provide  procedural  refresher  video  sequences  for  tasks  that 
are  not  commonly  performed  or  require  the  performance  of 
complex  procedures. 
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Appendix  A;  Publication  Structural  Overview 

The  present  Technical  Order  (TO)  set  is  hierarchical  in 
structure  with  three  distinct  levels  of  publications.  At 
each  level  of  the  hierarchy  the  information  provided  changes 
in  both  detail  and  style.  At  level  I  (GV)  general 
descriptions  of  the  overall  vehicle  and  systems  are  given  as 
well  as  lists  of  abbreviations,  symbols,  mathematical  signs 
used  (3:1).  The  GV  also  provides  warnings  and  safety 
guidelines  for  the  handling  of  hazardous  materials, 
identifies  potential  vehicle  hazards  and  describes 
maintenance  procedures  in  general  terms  (3:1). 

Level  II  publications  provide  more  specific  detailed 
information  based  on  individual  vehicle  systems.  System 
theory  and  principles  of  operation  are  explained  and  lists 
of  consumable  and  support  equipment  are  provided.  This 
lev^il  also  has  simplified  schematic  diagrams  for  both 
electrical  and  hydraulic  systems  (3:1). 

Job  Guides  are  located  at  Level  III  and  provide 
detailed  step  by  step  procedural  information  to  guide  the 
user  through  the  various  maintenance  and  repair  procedures 
associated  with  the  vehicle.  These  procedures  are  presented 
in  a  "function,  task,  step  sequence  for  each  LRU  of  a  system 
or  subsystem"  (3:1).  The  terms  function,  task  and  step  are 
defined  as: 

a.  Function  -  a  series  of  one  or  more  tasks  (21:8) 
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b.  Task  -  a  series  of  one  or  more  steps  that  lead  to 
the  completion  of  a  procedure  (21:8) 

c.  Step  -  a  "single  or  separate  maintenance  action" 
(21:8) 

The  three  publication  levels  are  detailed  in  Figure  7. 


GENERAL  VEHICLE 


LEVEL  I 


LEVEL  II 


Figure  7:  Technical  Order  Publication  Hierarchy 


I 


i 
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Analysis  of  the  type  of  information  at  each 
hierarchical  level  revealed  that  there  are  three  distinct 
information  groups.  These  are: 

a.  Authorative  -  gives  formal  authority  for  the 
performance  of  the  tasks,  specified  in  the 
publication  set,  on  the  vehicle. 

b.  Descriptive  -  describes  the  operation  and  function 
of  the  various  elements  of  the  vehicle,  equipment 
and  tools  to  be  used  anc  the  physical  location  of 
the  systems,  sub-systems  and  LRU's  on  the  vehicle. 

c.  Procedural  -  provides  step  by  step  instructions  on 
the  performance  of  the  specified  task. 

The  type  of  information  provided  differs  at  each  level 
with  the  following  groupings  identified: 

a.  LEVEL  I  -  Authorat ive/Descriptive/Procedural 

b.  LEVEL  II  -  Descriptive/Procedural 

c.  LEVEL  III  -  Procedural/Descriptive 

This  is  represented  graphically  in  Figure  8. 


Publication  Numbering  System 

Publication  numbering  system  compliance  requirements 
are  '•'^tailed  in  MlL-M-83495  and  require  all  manufacturers  to 
comply  with  this  standard.  This  standard  establishes  three 
different  numbering  systrms  which  result  in  all  publications 
for  a  specific  vehicle  being  linked  together  (21:7). 

These  number  systems  are  the  Technical  Order  number, 
system/Subsystem/Gub ject  Number  (S/S/SN)  and  the  Fault 


LEVEL  I 


AUTHORATIVE/ 

DESCRIPTIVE/ 

PROCEDURAL 


T 

1 

j 

LEVEL  11 

*7 

/ 

.  LEVEL  III 

DESCRIPTIVE/ 

PROCEDURAL 


PROCEDURAU 

DESCRIPTIVE 


Figure  8:  Publication  Content  Stratification 


Reporting/Fault  Indicator  (FRI/FI)  numbers.  These  systems 
are  graphically  illustrated  in  Figure  9. 

Under  this  numbering  method  each  system  and  subsystem 
is  given  a  unique  number  which  when  combined  provide  the 
required  indexing  reference  throughout  the  publication  set. 
Thus  any  user  can  easily  locate  needed  information  from  any 
publication  for  a  specific  system  or  subsystem  by  knowing 
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|— —  <T0)  1F-15C'2-2UO-60  1 

1  .  .  .  W««^n  ic]«n(i(i«r  <Air(;r«<l) 

F  . . .  Specific  type  aircraft  fFighter) 

15  Air  Force  model  dcsfo'^^tlon  acrks  number 
C  . .  Specific  model  tn  aeries  (could  also  de  A.  6,  or  0| 

2  .  Applicable  maintenance  level  (Org(ir>i:attort«i) 

20  Aircraft  system  covered  (Air  Co'idillontng) 

JG  .  Type  of  manual  (Job  Guide) 

60^  AIrcrall  subsysieni  (8i«ed  Air  Tenkperature  Control) 
1...  Martuai  vo'ume  numbe' 


(S/S/SNk21-602S 


AIrcfall  iyslem  (Air  CuniJItloning) 

O'  60^.  Aircraft  aubsyatom  (6l«cC  Air  Ternperalyre  Contro'l 


25  -  . &jb|acl  or  function  (narriorai.  Installation,  and 
oparalional  checkout  ol  Le*l  and  Right  Primary 
Heat  Exchanger  Ejector  Valves) 


- (FR/FI)  Fault  Cotfe  216061C1 

.  Alfciait  system (AirCondit  onlngl 


Alfcra't  subsystem (B'eed  Air  Tcmpeiature  Control) 


'  •  ■  ■  1  DIglls  BSSlgned  according  to  specific  m8lluncllon« 
C . . .  I  Identified 


Figure  9i  Publication  Numbering  Systam 

(19:7) 


only  its  S/S/SN  number  (21:7). 

This  approach  v/orks  well  within  the  current  top-down 
system/subsystem  publication  structure  as  the  amount  of 
referential  information  contained  in  the  manuals  could  not 
feasibly  be  held  in  one  publication  due  to  the  sheer  volume 
and  the  complex  indexing  that  would  be  required  to  support 
information  location. 


General  Service  Manual 

The  manual  analyzed  for  this  segment  of  the  research 
was  the  TO  1F-15E-2-32GS-00-1,  Landing  Gear  System  USAF 
Series  F-15E  Aircraft.  This  publication  contains  technical 
maintenance  instructions  for  the  F15E  aircraft  at  the 
system,  subsystem  and  sub-subsystem  level.  The  subsystem 
under  analysis  is  the  Nose  Landing  Gear  (NLG) . 
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The  GS  manual  contains  the  following  sections  that 
relate  specifically  to  the  NLG  subsystem: 

a.  Section  20  -  Nose  Landing  Gear  and  Doors 

1.  System  Functional  Description 
i.  Description 

ii-  Principles  of  Operation 

2.  Special  Maintenance  Requirements 

3 .  Nose  Landing  Gear  Pertinent  Data 

4 .  Consumable  Materials  List 

5.  Supplies  (consumable) 

6.  Support  Equipment  List 

7.  Test  Equipment 

8.  Special  Tools 

In  addition,  the  following  sections  also  contain  elements 
pertinent  to  the  NLG  subsystem: 

a.  Section  30  -  Extension  and  Retraction 

b.  Section  50  -  Steering 

c.  Section  60  -  Position  and  Warning 

The  NLG  references  in  the  GS  manual  were  concept  mapped 
to  highlight  relationships,  content  and  presentation  format. 
The  higher  level  concept  map  is  detailed  at  Figure  10  and 
the  lower  level  is  shown  at  Figure  11.  Examination  of  the 
map  reveals  that  the  content  meets  the  previous  assessment 
of  Descriptive/Procedural,  however  presentation  is  in  one  of 
three  formats: 
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Figure  10:  NLG  GS  Manual  Data  References  -  Level  0 


Figure  11:  NLG  GS  Manual  Data  References  -  Level  1 
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a.  Graphical  -  drawings  and  diagrams 

b.  Conceptual  -  written  descriptions 

c.  Tabular  -  data  is  present  in  column  format 

GS  Manual  Analysis 

Conceptual .  At  this  publication  level  the  conceptual 
information  presentation  is  in  the  written  verbose  form  and 
provides  the  user  with  a  technical  explanation  of  the  system 
and  its  general  theory  of  operation.  There  is  little 
attempt  made  to  link  the  conceptual  data  presentation  to 
either  the  graphical  or  tabular  representations.  Use  of 
technical  abbreviations  is  made  consistent  with  definitions 
of  LRU's,  systems  and  subsystems  provided  in  this  and  other 
publications,  h^'^^ever,  there  are  no  direct  references  made 
to  either  the  tabular  or  graphical  data.  The  user  must 
resort  to  the  graphical  in  order  to  identify  physical 
location  of  a  part  on  the  vehicle  and  then  cross  reference 
this  to  the  supporting  tabular  data  for  explanatory 
information.  At  this  level,  the  conceptual  data 
presentations  are  task  supportive  in  that  they  provide 
descriptive  data  not  directly  involved  in  task  performance. 

Tabular .  The  tabular  data  is  presented  in  columnar 
format  and  meets  two  distinct  roles: 

a.  Provide  descriptive  data  on  a  specific  LRU  or 

subsystem  identified  either  by  physical  location 
on  the  vehicle  or  in  alphabetical  order  by  its 
common  name.  Columns  provide  sufficient 
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information  to  enable  the  user  to  move  to  other 


publications  as  desired. 

b.  Provide  supportive  data  relative  to  ancillary 

equipment  and  supplies  that  support  the  LRU's  and 
proposed  maintenance  actions.  This  data  is 
provided  in  alphabetical  order  and  describes  uses 
and  applications  as  well  as  manufacturers  part 
numbers . 

Information  acquired  at  sub-para  a.  is  by  either  identifying 
a  specific  subsystem  or  LRU  on  the  physical  representation 
of  the  vehicle  or  looking  up  in  the  table  by  a  known  common 
name.  On  the  other  hand,  data  provided  at  sub-para  b.  is 
generally  knowledge  required  to  support  the  performance  of  a 
specific  maintenance  task,  thus  the  user  must  access  this 
prior  to  commencing  the  task.  Repetitions  of  table  entries 
were  identified  between  tables  in  different  sections,  in 
particular  the  Consumable,  Test  Equipment  and  Special  Tools 
tables . 

Graphical .  Graphical  data  presentations  are 
illustrations  of  the  various  systems,  subsystems,  sub¬ 
subsystems  and  LRU's  that  provide  the  user  with  a  visual 
reference.  There  are  four  levels  of  graphics  at  the  GS 
manual  level: 

a.  Level  1  -  detailed  diagrams.  This  graphic 

provides  the  user  with  a  detailed  illustrations 
supported  by  numbered  call-outs  referenced  to  the 
tabular  system  Component  Location  List.  This  is 
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Figure  12:  Level  1  Graphic  -  Nose  Landing  Gear 

(3; 20-3) 
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illustrated  in  Figure  12. 
b.  Level  2  -  Vehicle  location  diagrams.  These 
illustrations  are  simplified  graphics  that 
identify  the  specific  location  of  the 
subsystem/ sub-subsystem/LRU.  They  are  supported 
by  call-outs  to  detailed  diagrams  which  in  turn 
have  call-outs  to  the  aforementioned  Locations 
Listing.  This  is  illustrated  in  Figure  13. 


Figure  13:  Level  2  Graphic  -  Vehicle  Location  Diagram 

(3:30-15) 

c.  Level  3  -  Schematic  diagrams.  These  provide  both 
electric  and  hydraulic  schematics  for  the 
subsystem/sub-subsystem/LRU.  At  this  publication 
level  they  are  generalized,  however,  the  major 
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components  are  referenced  to  the  Parts  Location 
Listing.  This  is  illustrated  in  Figure  14. 
d.  Level  4  -  special  tool  illustrations.  These 
diagrams  are  self  explanatory  providing 
illustrations  of  special  tools  required  for  task 
performance.  See  Figure  15. 

Level  2  illustrations  also  describe  component  and  vehicle 
referential  views  from  the  front,  back,  left  side,  right 
side,  top  and  bottom.  Level  2  diagrams  are  repeated 
throughout  the  publication  and  maintain  size  and,  depending 
on  the  view,  aspect.  In  relation  to  task  performance  the 
graphical  representations  at  the  GS  manual  level  fulfill  a 
part  identification  and  location  reference  role. 
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Figure  15:  Level  4  Graphic  -  Special  Tools  Illustration 

(3:30-30) 
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Appendix  B:  Model  Evaluation  Questionnaire 


SHADM  MODEL  EVALUATION 
QUESTIONNAIRE 

After  using  the  SHADM  model  program  please  indicate  on  the  scales  your  feelings  about  the  following 
statements.  Where  you  score  a  statement  below  5  please  provide  supporting  comments  for  your  score. 

User  Interface 


STRONGLY 

DISAGREE 


STRONGLY 

AGREE 

I 

10 


1. 


The  program  is  easy  to  start. 


Comments: 


STRONGLY 

DISAGREE 


I 


0  1  2  3  4  5  6 

2.  The  use  of  the  program  is  not  difficult  to  understand. 

Comments: 


STRONGLY 

AGREE 

I 

10 


STRONGLY 

DISAGREE 

(III) 

0  12  3  4 

3.  Navigating  within  the  system  is  easy. 

Comments: 


STRONGLY 

AGREE 

I 

10 


STRONGLY 

DISAGREE 


STRONGLY 

AGREE 

I 

10 


4.  The  direct  addressing  capability  made  it  easier  for  the  experienced  person  to  use  the 

system. 

Comments: 


STRONGLY 
DISAGREE 

I 

0 

5.  Overall  the  program  is  user  friendly. 

Comments: 


STRONGLY 

AGREE 

I  I 

9  10 
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STRONGLY 
AGREE 

I  I  I  I  I  I 

5  6  7  8  9  10 

6.  The  textual  presentation  is  easy  to  read. 

Conments: 


STRONGLY 

DISAGREE 


STR(  _LY 
AGREE 

till 

7  a  9  10 

7.  The  menus  are  laid  out  logically  and  are  easy  to  read. 

Comments: 


STRONGLY 

DISAGREE 


STRONGLY 
AGREE 

I 

10 

a.  The  graphic  to  text  format  of  the  task  procedure  step  presentation  is  best. 

Comments: 


STRONGLY 

DISAGREE 

I  I  I  I  I  I  I  I  I  I 

0123456789 


STRONGLY 
AGREE 

I  I 

9  10 

9.  The  text  to  graphic  format  of  the  task  procedure  step  presentation  is  best. 

Comments: 


STRONGLY 

DISAGREE 


STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  I  I  I  I  I  I  I  I  I  I 

0123456789  10 

10.  The  use  of  different  windowing  techniques  makes  the  information  easier  to  use. 


Comments: 


STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  I  I  I  I  I  I  I  I  I  I 

0123456789  10 

11.  The  data  is  segmented  into  logical  groups. 

Connents: 


STRONGLY 
AGREE 

I  I  I 

8  9  10 

12.  The  structure  of  the  information  is  good  and  facilitates  easy  use. 

Comments: 


STRONGLY 

DISAGREE 

I  I  I  I  I  I  I  I 

0  1  2  3  4  5  6  7 


STRONGLY 
AGREE 

I  I  I 

8  9  10 

13.  The  graphic  menu  choice  will  make  it  easier  for  new  users  to  use. 

Comments: 


STRONGLY 

DISAGREE 

I  I  I  I  I  I  I  I 

0  1  2  3  4  5  6  7 


STRONGLY  STRONGLY 

DISAGREE  AGREE 


14,  Navigating  through  the  information  is  made  easier  by  its  structure  and  layout. 
Comments: 


STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  I  I  I  I  I  I  I  I  I  I 

0123456789  10 

15.  The  links  between  the  various  information  nodes  are  logical. 

Comments: 
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General 

STRONGLY  STRONGLY 

DISAGREE  AGREE 

I  I  I  I  I  I  I  I  I  I  I 

0123456789  10 

17.  The  package  meets  the  needs  of  the  user. 

Comments; 


STRONGLY 
AGREE 

I  I  I  I  I 

6  7  8  9  10 

18.  I  would  use  this  system  in  a  workshop  environment. 

Comments: 


STRONGLY 

DISAGREE 


STRONGLY 
AGREE 

I  I 

9  10 

19.  I  would  use  this  system  in  a  Systems  Project  Office  to  manage  publications. 

Comments: 


STRONGLY 

DISAGREE 

I  I  I  I  I  I  I 

0  1  2  3  4  5  6 

20.  The  system  will  be  accepted  by  the  technician  end  user. 
Comments: 


STRONGLY 

AGREE 

I  I 

9  10 
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Please  feel  free  to  address  any  specific  strengths  or  weaknesses  you  observe  in  the  package  but  were 
not  addressed  specifically  by  the  questionnaire.  Use  the  space  below  or  attach  additional  sheets  as 
required. 

Conments: 
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Appendix  C:  Questionnaire  Responses 
The  SHADM  model  was  prototyped  using  the  HyperWriter 
hypermedia  software  package.  The  prototype  was  validated  by 
eight  personnel  in  unsupervised  testing.  The  following  is  a 
compilation  of  their  assessments  and  recommendations. 
Question  1:  The  program  is  easy  to  start. 

Scores:  6(2),  7(2),  9(2),  10(2). 

Comments:  1.  Warrants  10  if  you  turn  computer  on  and  it 

automatically  takes  you  to  user  login  and 
password . 

2.  With  users  guide  available  for  knowledge 
of  executable  commands. 

3.  A  more  dedicated  interface  would  obviously 
be  used  in  a  real  application. 

Question  2:  The  use  of  the  program  is  not  difficult  to 

understand . 

Scores:  5(1),  6(2),  8(1),  9(4). 

Comments:  1.  Program  is  almost  self  explanatory. 

2.  At  least  as  far  as  the  user  interface 
makes  the  program  easy  to  use. 

3.  After  very  little  experimentation  easy  to 
understand . 

4.  Some  of  the  concepts  would  need  to  be 
taught  to  anyone  not  computer  orientated. 

Question  3:  Navigating  within  the  system  is  easy. 

Scores:  6(1),  8(3),  9(3),  10(1). 

Comments:  1.  Pull  down  windows  are  great. 
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Question  4 : 


Scores : 
Comments : 

Question  5: 
Scores : 
Comments : 


Question  6: 
Scores : 
Comments : 


Question  7: 


Scores : 


2.  Mouse  application  very  good. 

The  direct  addressing  capability  made  it 
easier  for  the  experienced  person  to  use  the 
system. 

8(1),  9(6),  Not  answered  (1). 

1.  Warnings,  Cautions  should  always  be 
present. 

Overall  the  program  is  user  friendly. 

7(1) ,  8(1) ,  9(3) ,  10(3) . 

1.  Very  friendly. 

2.  For  the  technician  to  use  as  a  T.O. 
(publication) ,  may  not  be  if  you  mean  use  of 
Hypermedia  to  program  publication  into 
format. 

The  textual  presentation  is  easy  to  read. 

4(1) ,  5(1) ,  7(1) ,  9(4) ,  10(1)  . 

1.  Blue  background  was  a  problem,  I  know  you 
are  looking  at  this  also,  some  numbers  and 
letters  were  difficult  to  read  (blue 
background  again) . 

2.  Black  text  on  blue  background  difficult 
for  this  reviewer,  some  displays  were  black 
text  with  art  on  white  background  -  better. 
The  menus  are  laid  out  logically  and  are  easy 
to  read. 

8(3),  9(5). 
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Comments : 


Question  8: 


Scores : 


Comments ; 


Question  9: 


Scores : 


Comments : 


1.  Being  familiar  with  83495  Mil  spec  may 
have  biased  my  perspective. 

2.  I  would  block  out  the  menu  'buttons'  to 
the  same  length  to  make  it  tidier. 

The  graphic  to  text  format  of  the  task 
procedure  step  presentation  is  best. 

2(1),  5(2),  7(1),  8(1),  9(1),  10(1),  Did  not 
answer  ( 1) . 

1.  Easier  to  use  text  to  graphics.  Having 
both  on  screen  would  be  beneficial. 

2.  It  would  be  more  simplistic  if  it  were  a 
split  screen. 

3.  Both  text-graphics  and  graphics  to  text 
are  appropriate;  in  certain  circumstances  one 
is  better  to  use  than  the  other  and  vice 
versa . 

The  text  to  graphic  format  of  the  task 
procedure  step  presentation  is  best. 

2(1),  5(2),  6(1),  7(1),  9(1),  Did  not  answer 
(2)  . 

Same  as  for  Question  8. 

1.  I  like  the  idea  of  being  able  to  see 
what's  going  to  happen.  A  picture  is  worth  a 
1000  words.  The  best  thing  that  could  happen 
is  having  the  text  listed  next  to  the  drawing 
and  maybe  even  a  video  option. 
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Question  10: 

Scores: 
Comments : 


Question  11: 
Scores : 
Comments : 


Question  12: 

Scores : 
Comments : 


2 .  Text  and  graphics  should  be  on  screen 
together  without  text  covering  text. 

The  use  of  different  windowing  techniques 
makes  the  information  easier  to  use. 

7(2)  ,  8(3)  ,  9(3)  . 

1.  Windows  keep  all  pertinent  information  at 
the  technicians  fingertips. 

2.  However,  as  you  have  done  in  some  cases, 
wherever  there  is  a  reference  to  a  figure  or 
procedure,  use  a  window  to  view  this  figure, 
procedure.  Not  all  links  have  been  included 
time  constraints  probably  prevented  you  from 
including  this  feature  in  all  cases  where 
required. 

The  data  is  segmented  into  logical  groups. 
6(1) ,  7(2) ,  8(3) ,  9(2) . 

1.  The  data  available  was  segmented 
logically. 

2.  Only  lack  of  user  experience  prevents  my 
scoring  higher. 

The  structure  of  the  information  is  good  and 
facilitates  easy  use. 

6(1) ,  7(3) ,  9(4) . 

1.  Again,  what  was  available  was  excellent. 

2.  But  watch  for  occasions  where  multiple 
screens  are  desired,  may  have  troops 
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Question 

Scores : 
Comments : 


Question 

Scores : 
Comments : 


Question 

Scores : 
Comments 


producing  hard  copies  to  work  in  concert  with 
display  info. 

13:  The  graphic  menu  choice  will  make  it  easier 

for  new  users  to  use. 

7(1)  ,  8(2)  ,  9(3)  ,  10(2) . 

1.  Quality  of  graphics  needs  to  be  enhanced. 
Once  this  is  accomplished  the  graphic  menu 
should  ease  the  utility  of  SHADM. 

2.  This  application  OK  but  several  other 
tasks  not  with  the  32-20-xx  series  are  in  N- 
W-W  e.g.  ASP  Panel  R&I  31-50-10.  Window 
needed  for  all  items  in  NWW  that  can  be 
accessed  via  graphics  menu. 

14:  Navigating  through  the  information  is  made 

easier  by  its  structure  and  layout. 

6(2) ,  7(2)  ,  8(1) ,  9(3)  . 

1.  Information  should  be  able  to  move  about 
the  screen. 

2.  Understanding  the  <green>  <blue>  are 
links.  (Users  guide) . 

3.  Yes,  but  'easier'  than  what?  Technicians 
are  very  used  to  hard  copy. 

15:  The  links  between  the  various  information 

nodes  are  logical. 

7(1) ,  8(4)  ,  9(2) ,  10(1) . 

1.  What  was  available  was  logical. 


96 


Question 

Scores : 
Comments ; 

Question 
Scores : 
Comments 


Question 

Scores : 
Comments 


2.  Once  again,  I  suspect  a  higher  score  is 
deserved  but  more  experience  is  needed, 

16:  The  links  are  correctly  made  and  facilitate 

easy  use  of  the  system. 

6(1)  ,  7(1) ,  8(4)  ,  9(1) ,  10(1) . 

1.  My  limited  knowledge  of  the  aircraft 
system  was  a  disadvantage,  however  I  didn't 
see  any  obvious  links  that  were  incorrect. 

17:  The  package  meets  the  needs  of  the  user. 

3(1) ,  5(1) ,  7(3)  ,  8(1) ,  9(1) ,  10(1)  . 

1.  This  package  needs  to  be  MIL-SPEC 'd. 

2.  Good  for  in-shop  use.  Now  it  needs  to  be 
able  to  be  used  on  the  flight-line. 

3.  For  in  shop  use  with  this  configuration 
hardware.  Flight-line  use  may  need  hand  held 
unit. 

4.  Subject  to  possibility  of  reguiring 
simultaneous  access  to  >  1  page  at  a  time, 
such  as  for  circuit  diagrams  in  f^ult  finding 
rather  than  scheduled  maint.  functions.  Also, 
might  need  package  for  fault  finding  needs. 

5.  Must  be  tested  by  a  true  user  to  answer 
this  I  suspect  so. 

18:  I  would  use  this  system  in  a  workshop 

environment . 

5(1)  ,  6(1)  ,  7(1)  ,  9(3)  ,  10(2)  . 

1.  If  MIL-SPEC'd. 
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Question  19; 

Scores: 
Comments : 


Question  20: 

Scores : 
Comments : 


2.  Excellent  for  workshop. 

3.  Subject  to  comment  3  at  question  17  and 
comment  2  question  12. 

I  would  use  this  system  in  a  Systems  Project 
Office  to  manage  publications. 

0(1)  ,  4(1)  ,  7(1)  ,  8(1)  ,  9(1)  ,  10(3)  . 

1.  There  wouldn't  be  a  practical  use  in  a  SPO 
since  we  use  partial  manuals,  35%,  70%. 

2.  Only  in  digital  data  format.  That  the  only 
way  changes  to  the  data  could  be  made  and 
sent  to  the  user. 

3.  Providing  amendments,  including  drawing 
changes,  could  be  readily  incorporated. 

4.  Keeping  publications  current  with 
engineering' data  not  function  of  SPO, 
contractor  responsibility.  Management  of 
publications  program  SPO  responsibility, 
don't  see  application  to  aid  in 
administrative  management  of  full 
publications  program. 

The  system  will  be  accepted  by  the  technician 
end  user. 

2(1),  5(1),  6(1),  7(1),  8(1),  9(2),  10(1). 

1.  The  hardware  needs  to  be  ruggedized  and 
environmentally  tested. 

2.  This  should  be  an  acceptable  system  for 
the  flight-line  supervisor  and  mechanic. 
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Being  familiar  with  computer  operation  would 
enhance  my  evaluation  of  this  system.  An  hand 
held  device  for  flight-line  use,  rugged 
enough  would  be  accepted  in  the  flight-line. 

3.  There  will  be  a  great  deal  of  user 
resistance  and  the  outcome  will  depend  on  the 
practicality  and  reliability  of  the 
associated  hardware. 

General  Comments 

1.  Good  System.  Easy  to  use  and  very  easy  to  move  through. 
Great  for  inshop  use.  Need  to  develop  a  way  for  flight-line 
use  without  paper. 

2.  How  much  growth  memory  is  there  available. 

3.  How  are  checklists  handled. 

4.  Any  warranty  or  time  change  recognition  in  database. 

5.  Warnings,  Cautions  need  to  be  called  out  in  text, 
before  procedure  is  performed. 

6.  When  more  than  one  page  of  text  is  presented  on  the 
screen,  especially  numerous  pages,  adjust  the  software  so 
that  one  can  go  from  page  to  page  rather  than  forwarding  one 
line  at  a  time. 

7 .  How  do  I  know  which  LRU  is  what 

Installation/Removal/Rigging.  Put  link  to  LRU  window. 

8.  Nothing  of  big  importance  but  I  prefer  the  listings  to 
go  down  the  columns  in  sequence  rather  than  across  columns. 

9.  Like  the  screen  selections  at  the  bottom  rather  than 
top  so  I  don't  have  the  listings  block  the  selections. 
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10.  USAF  MIL  SPEC  MIL-M-83495  and  MIL-M-38784  require 
warnings/cautions  to  precede  steps  that  they  apply  to. 

11.  There  are  106,000  pages  of  text  and  art  in  the  F-15E  JG 
set.  How  much  storage  space  is  required  on  computer  to 
handle  these  units/pages. 

12.  The  more  information  that  is  stored  in  computers,  what 
is  the  effect  on  system  speed. 

13.  As  we  have  discussed,  I  feel  the  key  to  it  all  will  be 
lie  in  the  practicality  of  the  hardware  needed  to  run  the 
system.  It  must  be  reliable  and  user  friendly. 

14.  The  potential  for  the  monitoring  and  management  of 
resources  is  enormous,  and  may  even  outweigh  the  more 
obvious  benefits  associated  with  document  maintenance. 

15.  Considerably  more  work  is  required  to  answer  your 
queries  about  the  real  use  of  the  system  and  the  logical 
layout  of  the  structure  to  meet  user  needs.  The  likes  of  you 
and  I  in  an  office  environment  can  never  anticipate  the  true 
needs.  However,  I  think  it  is  a  path  we  must  go  down  and  a 
trial  of  the  system  in  the  real  world  (possible  after  some 
simulation  exercises)  would  be  very  worthwhile. 

16.  Biggest  problem  I  see  is  how  to  include  circuit 
diagrams  and  other  schematic  diagrams  required  for  fault 
diagnosis  of  electrical  and  electronic  sub-systems.  A 
corollary  of  this  is  that  technicians  might  be  included  to 
produce  hard  copies  of  these  pages  so  that  he  may  access  >  1 
page  simultaneously.  Unfortunately,  such  pages  are  not 
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subject  to  amendment  and  may  therefore  contribute  to 
maintenance  errors  if  relied  upon. 

17.  Also,  in  some  steps,  such  as  the  steps  required  to 
install  LRU  32-20-18,  some  of  the  steps  in  the  30-steps 
process  were  not  included.  As  I  do  not  know  the  particular 
procedure,  I  don't  know  if  you  left  them  out  because  the 
book  did,  or  you  didn't  have  time  to  include  them.  Whatever 
the  reason,  omission  of  procedural  steps  left  me  wondering 
about  those  steps  I  haven't  undertaken,  and  also  inspired 
some  doubt  as  to  whether  or  not  I  was  using  the  package 
correctly. 

18.  Overall,  very  impressed  with  product.  I  see  further 
applications  for  Systems  Engineers  in  staff  positions.  Also, 
this  type  of  tool  should  be  made  available  as  part  of  the 
system  deliverable  (at  least,  if  not  this  system  exactly, 
then  some  system  following  the  broad  principles) . 
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Appendix  D;  Prototype  Test  Question  Analysis 


This  Appendix  provides  question  by  question  analysis  of 
the  Prototype  testing  results. 

Question  1  Ease  of  starting  the  model  was  rated  as 
very  good  with  an  average  score  of  8  from  a  range  of  6  to 
10.  Individual  comments  were  addressed  to  peripheral 
issues,  which,  while  not  directly  pertinent  to  the 
underlying  model  design,  are  considered  relevant  to  overall 
package  presentation  and  ultimate  user  acceptance. 

Question  2  Program  understanding  was  assessed  as  good 
with  an  average  score  of  7.625  from  a  range  of  5  to  9. 
Comments  were  in  the  most  part  complimentary  of  the  system 
although  one  user  addressed  the  issue  of  computer  literacy 
of  end  users  and  the  need  to  provide  adequate  user 
documentation.  The  need  for  appropriate  and  timely 
training,  and  documentation  in  the  form  of  a  User  Guide  is 
seen  as  an  implementation  strategy. 

Question  3  Navigation  within  hypertext  documents  is 
considered  a  significant  problem  area.  Due  to  the  non¬ 
linear  layout  and  proliferation  of  nodes  and  links,  it  is 
very  easy  for  users  to  become  lost  or  disoriented.  This 
aspect  of  the  model  was  rated  as  very  good  with  an  average 
score  of  8.375  and  a  range  of  6  to  10.  Amplifying  comments 
were  limited  to  complimentary  statements  about  individual 
navigation  aids  provided  in  the  model  and  developed  no 
significant  issues  worthy  of  further  consideration. 
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Question  4  The  model  offered  the  user  two  menuing 
options,  firstly  a  graphic  selection  where  the  desired 
system  was  selected  from  an  aircraft  illustration.  This 
option  was  provided  for  the  inexperienced  user  and  took  the 
user  through  a  number  of  menu  levels  to  the  desired 
function.  Direct  addressing  was  the  second  option  provided 
and  this  was  directed  at  the  experienced  user,  allowing  a 
menu  choice  that  led  directly  to  the  desired  function, 
bypassing  the  other  menu  layers.  The  evaluation  panel  rated 
this  aspect  of  the  model  as  very  good  with  an  average  score 
of  8.85  and  a  score  range  of  8  to  9.  This  indicated  that 
the  high  rating  assessment  was  shared  by  all  users.  One 
user  did  not  provide  a  response  to  this  statement.  Only  one 
comment  was  received  on  this  aspect  of  the  model  and  that 
was  complimentary,  offering  nothing  to  model  enhancement. 

Question  5  In  overall  user  friendliness  the  users 
rated  the  model  as  very  good  giving  it  an  average  score  of  9 
and  a  range  of  7  to  10.  Amplifying  comments  were 
complimentary  and  added  nothing  to  further  model 
development . 

Question  6  Readability  of  textual  presentation  rated 
an  average  score  of  7.75  from  a  range  of  4  to  10.  This  wide 
range  of  evaluations  was  supported  by  comments  indicating 
dissatisfaction  with  the  graphics  display  colors,  black 
illustration  on  blue  background.  This  unfortunately 
resulted  from  a  conflict  between  the  color  palette  of  the 
graphics  package  and  the  display  palette  of  HyperWriter,  a 
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matter  that  has  been  addressed  to  NTERGAID.  Users  indicated 
that  the  preferred  display  colors  for  graphics  are  black 
illustrations  on  white  backgrounds,  as  specified  in  the 
SHADM  model. 

Question  7  Menu  structure  and  layout  received  both 
favorable  comment  and  a  very  good  rating  with  an  average 
score  of  8.625.  Individual  comments  related  to  the  sizing 
of  link  buttons  on  menus,  expressing  a  preference  for  a 
standard  size  to  create  display  uniformity.  This  is 
acknowledged  and  will  be  included  in  the  model. 

Question  8  With  an  average  score  of  6.57  the  graphic 
to  text  display  format  was  only  rated  by  the  users  as  good. 
Opinions  regarding  this  format  were  widely  varied,  indicated 
in  the  score  range  of  2  to  10.  One  user  did  not  respond  to 
this  question.  Qualifying  comments  indicate  that  the 
preference  is  to  have  both  the  text  and  the  graphic  on  the 
screen  simultaneously  and  in  such  a  manner  that  neither 
encroaches  on  the  other  Additionally,  users  indicated  that 
both  graphics  formats  may  be  appropriate  depending  upon  the 
circumstances.  One  option  is  to  provide  both  formats  within 
the  model,  although  this  would  incur  additional  programming, 
memory  and  data  storage  overhead,  however,  this  would  appear 
a  negligible  consideration  when  the  improved  user  assessed 
utility  and  friendliness  is  taken  into  account. 

Question  9  The  graphic  presentation  option  of  text  to 
graphic  was  less  well  liked  by  the  users  than  the  text  to 
graphic  display.  The  average  score  was  5.67  which  rated  it 
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as  acceptable.  The  score  range  was  from  2  to  9  with  two 
users  not  providing  responses.  The  individual  comments  were 
of  a  similar  nature  to  those  provided  for  question  8  and 
analysis  is  unchanged. 

Question  10  Users  appear  satisfied  with  the  use  of 
different  windowing  techniques  and  this  is  reflected  in  the 
high  average  score  of  8.125,  a  rating  of  very  good.  No 
additional  meaningful  input  was  provided  through  supporting 
comments . 

Question  11  Logical  segmentation  of  data  was 
classified  as  good  with  a  users  average  score  of  7.75  and 
score  range  of  6  to  9.  Supporting  comments  justifying  this 
low  rating  were  not  provided. 

Question  12  Question  12  examined  the  structure  of  the 
data  provided.  Users  only  rated  the  structure  as  good  with 
an  average  score  of  7.875,  while  overall  scores  ranged  from 
6  to  9 .  Concern  was  expressed  that  while  the  data  was 
structured  well,  there  may  be  a  necessity  for  users  to  view 
multiple  screens,  and,  without  a  way  of  moving  quickly 
between  them,  users  may  be  induced  to  provide  hardcopy  to 
meet  a  perceived  system  deficiency.  The  result  of  this 
would  be  that  these  hard  copies  are  retained  for  future  use 
and  because  the  paper  copies  would  not  be  updated  by 
publication  amendment  action,  the  user  would  be  relying  on 
outdated  data.  This  is  an  extremely  serious  problem  which 
can  be  solved  by  including  a  facility  in  the  model  that  is 
available  from  the  HyperWriter  software.  HyperWriter 


105 


provides  a  facility  for  the  user  to  implement  place  markers 
which  enable  the  system  to  remember  specific  window 
locations.  This  would  allow  the  user  to  quickly  move 
between  windows  using  short  keystroke  combinations  (16 : 30) . 
The  requirement  for  this  capability  will  be  written  into  the 
model  specifications  based  on  this  recommendation. 

Question  13  This  question  addressed  the  graphical  menu 
provided  for  new  users.  Participants  rated  it  as  very  good 
with  an  average  score  of  8.75  from  a  range  of  7  to  10. 
Individual  comments  addressed  the  quality  of  graphics  used 
and  the  need  to  introduce  an  additional  menu  layer  after  the 
initial  aircraft  display.  This  is  to  ensure  that  all 
subsystems  and  sub-subsystems  can  be  displayed  and  made 
available  for  user  selection.  This  is  a  valid  comment  and 
can  be  implemented  using  the  existing  three  tiered  menu 
structure  already  existing  in  the  model. 

Question  14  Question  14  sought  assessment  of  the 
navigation  capabilities  of  the  system  with  particular 
emphasis  on  ease  of  movement  through  the  document. 

Individual  scores  were  evenly  spread  over  the  range  6  to  9 
and  the  overall  average  was  7.625  which  rated  the  navigation 
aspects  as  only  being  good.  Individual  comments  addressed 
movement  through  data  page  at  a  time,  a  facility  that  is 
already  available  on  the  tested  system  but  not  identified  in 
the  initial  briefing.  This  aspect  leads  into  the  second 
comment  on  the  need  for  a  users  guide.  This  need  is 
recognized  and  in  a  field  application  of  this  model  a  users 
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guide  and  training  course  should  be  provided  to  end  users. 
One  user  commented  on  the  end  users  predilection  for  the 
current  hardcopy  system,  a  comment  that  was  further 
developed  by  the  user  at  question  20.  This  comment  was 
examined  in  greater  detail  in  the  analysis  of  question  20. 

Question  15  Logical  link  evaluation  was  the  subject  of 
question  15  and  the  users  rated  this  aspect  as  very  good 
with  an  average  score  of  8.375  from  a  range  of  7  to  10. 
Comments  provided  were  of  a  confirmatory  nature  and  added 
nothing  to  further  model  development. 

Question  16  Question  16  addressed  the  correctness  of 
the  links  and  how  easy  they  made  the  system  to  use.  The 
range  of  scores  was  wide  from  6  to  10  and  the  average  score 
of  8  gave  the  linking  strategy  a  rating  of  very  good.  Qnly 
one  amplifying  comment  was  offered  which  questioned  an 
individual  users  ability  to  properly  evaluate  this  aspect  of 
the  model  design  due  to  lack  of  aircraft  data  knowledge. 

This  is  considered  a  valid  comment,  however,  it  offered 
nothing  in  the  way  of  further  model  development. 

Question  17  This  question  was  designed  to  determine 
how  well  the  package  met  the  users  needs.  With  an  average 
score  of  7,  this  aspect  only  rated  as  good.  Examination  of 
individual  responses  indicated  that  the  severest  criticism 
came  from  a  technician  publication  manager  who  expressed 
concern  over  the  failure  of  the  tested  system  to  meet  MIL- 
Specif ications.  In  its  current  presentation  style  he  rated 
it  as  unacceptable.  Given  the  requirement  in  the  model 
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specifications  for  compliance  with  MIL-Specif ications,  no 
further  action  will  be  taken  on  this  recommendation.  Other 
comments  addressed  the  ability  of  the  package  to  perform  in 
the  flight-line  environment  and  particularly  highlighted 
hardware  capabilities  and  requirements.  This  area  is  beyond 
the  scope  of  this  study  however  it  is  an  important 
consideration  for  future  studies  and  will  be  discussed  in 
the  recommendations  section.  Other  comments  identified  the 
perceived  need  for  users  to  be  able  to  access  more  than  one 
page  at  a  time  during  diagnostic  procedures  requiring  use  of 
wiring  diagram  or  hydraulic  schematics  that  involve  multiple 
displays.  This  issue  was  previously  discussed  at  question 
12  with  the  implementation  of  marker  strategies. 

Question  18  Question  18  sought  to  determine  how  useful 
the  users  perceived  the  system  would  be  in  a  workshop 
environment.  Overall,  users  gave  this  aspect  of  the  model  a 
very  good  rating  with  an  average  score  of  8.125  from  a  range 
of  5  to  10.  Individual  comments  were  related  to  the 
packages'  non-compliance  with  military  documentation 
specifications,  this  user  rating  it  as  a  5  in  present 
format.  This  issue  was  previously  discussed  at  question  17. 

Question  19  This  question  was  designed  to  follow  on 
from  question  18  to  identify  the  environments  in  which  the 
users  perceived  the  model  as  being  usable.  Use  in  a  Systems 
Project  Office  was  rated  as  good  with  an  average  score  of 
7.25  from  a  range  of  0  to  10.  Amplifying  comments  indicated 
that  SPO's  are  not  normally  involved  in  the  day  to  day 
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preparation  and  use  of  publications.  The  high  rating  is 
explained  by  users  not  involved  in  or  knowledgeable  about 
the  function  and  role  of  the  SPO  in  publications  management 
interpreting  it  as  the  central  publications  manager 
responsible  for  producing  and  updating  documentation  prior 
to  distribution  to  units.  This  hypothesis  is  supported  by 
the  high  scores  allocated  by  the  foreign  engineers.  Thus, 
while  the  average  score  rates  this  function  as  good 
indications  are  it  is  biased  and  therefore  not  acceptable. 

Question  20  The  final  question  in  the  test  addressed 
whether  the  user  believed  that  the  model  would  be  accepted 
by  the  technician  end  user.  While  this  called  for  the  offer 
of  an  opinion  it  was  designed  to  call  on  the  users 
experience  within  his  professional  field,  exposure  to  the 
package  and  knowledge  of  the  existing  system.  Non  USAF 
testers  and  non  F-15E  SPO  personnel  were  expected  to  draw  on 
knowledge  of  their  own  maintenance  publication  system  and 
its  use.  Overall  this  aspect  of  the  test  was  rated  as  good 
with  an  average  score  of  7.  However,  the  range  of  scores 
was  from  2  to  10  indicating  a  wide  disparity  in  opinions. 
Individual  comments  mainly  revolved  around  providing  the 
requisite  hardware  that  would  meet  the  end  users  needs  to 
deliver  information  and  be  capable  of  surviving  the  rugged 
working  environment.  The  one  score  of  2  was  given  by  a 
foreign  engineer  who  believed  that  there  would  be 
significant  user  resistance  to  the  system,  dependent  on  the 
"practicality  and  reliability  of  the  associated  hardware". 
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While  this  aspect  is  acknowledged  as  a  potential  problem 
area  it  is  not  considered  of  relevance  to  this  specific 
test.  Future  research,  however,  should  address  this  aspect 
prior  to  any  implementation. 
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